"When a genre novel (or a literary novel) reaches beyond its "traditional" audience and is bought by people who aren't normally fans of that variety of fiction, it can be said to be mainstream fiction...
Stephen King, a horror novelist, might also be considered a mainstream novelist by virtue of his huge popularity and his ability to attract readers who wouldn't normally read horror novels written by less well known writers.
The English Patient, a literary novel, also became a mainstream novel when it was bought by people who wouldn't normally buy literary novels and it sold in huge numbers (helped, no doubt, by the Oscar-winning film adaptation).
So that is the first definition and, let's be honest, one that isn't particularly helpful to you as you try to decide what type of book you are going to write!
After all, if your genre novel or literary novel becomes hugely popular and gets classified as "mainstream" by the publishers and booksellers, it still remains a genre novel or a literary novel...
It will just have a bigger marketing budget behind it.
It will be displayed more prominently in the bookshops.
And it will make your bank account grow fatter, of course!"
http://www.novel-writing-help.com/mainstream-fiction.html
I found this link by entering this query: define mainstream fiction.