I wouldn't bend this whole bit about formula too hard. Not on any genre. For example, a genre which is, by definition, more "free form" is formulaically so.
Anyway, to give you a heads up on the "formula" of horror, here are two examples of pure horror:
In the Exorcist, more people died in the making than in the movie. Did Reagan actually die at the end when he jumped off the top story? I don't know for sure, but I don't think so. Rosemary's Baby, another cornerstone for the horror genre, had no deaths directly related to the plot, and although I can't say for certain, I'm not sure anyone died in that film.
But I don't think that the main character living or dying before the end of the story is a trope of either genre, so let's not get carried away.
However, one thing to pay attention to is how much "the supernatural" plays a part in the role of the plot. If a world is fantastic and we know this upfront, then what we consider supernatural (the existence of magic, for example) is not considered supernatural in that world-- this would lean the story closer to fantasy than horror. However, if the world seems realistic/conventional to some degree and "the supernatural" is then imposed upon the natural in the events of the story, the plot reflects more the horror aspect [Biology of Horror, Morgan, 2002]. This is why Anne Rice's vampire novels are considered Dark Fantasy rather than horror-- because the existence of the vampire is the obvious reality of the characters in the story, and we as the readers know this immediately. But, Count Dracula, on the otherhand, was discovered to be supernatural, and his existence as that monster was imposed upon a more conventional-- though atmospheric-- sense of reality.
AMC