Hi, BlueTexas!
Well, "mainstream" isn't really a definition of genre so much as a definition of market. There are only two types of market for the purpose of popular fiction: mainstream and niche. Mainstream fiction includes genre and literary and is marketed to the general populace. Niche fiction is generally sold to small groups of people with specialized tastes. For example, a novel based on a limited run video game is probably targeted to the niche market of the players of that game.
I think what you mean is that you're trying to decide whether your book is Horror or some other category. That sort of depends on the nature of the book. Here are some definitions that might help you decide:
Horror: The plot must contain earthly or otherworldly threats to the hero or heroine (H/h) that end in death or torture of third parties and continual threat of death to the H/h. The goal is to frighten the reader and often this means non-redeeming madness or death. Horror need not have a "Happily Ever After" (HEA). In other words, in horror, the H/h may die and the bad guys win. An example of this would be Stephen King's The Shining. Several key characters don't make it out alive.
Thriller: The plot may be intellectual or physical in nature, but the threat of world domination, world destruction or some other major catrostophe that involves people beyond the H/h or their friends/family must be a primary plot device. An example of this would be Enemy of the State. Normally, in a thriller, the primary characters live through the experience and correct the threat.
Suspense: Suspense differs from thrillers in the level of action. Suspense novels generally have less action involved, even though there is still a risk of harm to the H/h. Rear Window is a classic suspense.
Your book might also be categorized as fantasy or science fiction, depending on the elements.
Does that help?
