I've always considered the two to be the same, and I'm trying to figure out which category my WIP would fit into.
(Apologies if this has already been asked!)
There isn't any way I could tell you which of the two categories (thank you for using the right word!) your WIP would fit into without reading it. However, I can at least offer a description of most fiction categories that has been reviewed, revised and generally agreed upon by many AW's. Click on the link in my signature below that reads:
What Genre (the wrong word)
is my writing? and scroll down to post #40.
There is also one other distinction between the two. It has been suggested that some publishers may not be
as willing to consider 'modern literary fiction' as they would be with 'mainstream/contemporary' works. I don't have any proof of this myself... but it may be worth 'fine tuning' your descriptive wording to be more in line with what each individual publisher is actively seeking.
You just named three different things. Mainstream means any book, of any type, and even of any genre, that captures the mainstream audience. Stephen King and J. K. Rowling are both mainstream. Contemporary just means stories set in the present. Literary means just what it says. You need to read the more famous literary writers to understand what this is.
A mainstream fiction novel may be a book from any category or genre. The term 'mainstream' describes a book that has been published and somehow or another managed to gain popularity and sales from a large, general (or mainstream) audience. If it was a book written in form of a horror story per the rules of that genre, then it is still horror fiction - and will be considered horror fiction long after the sales have dropped off. So, seemingly any book from any category can become a mainstream novel since mainstream is all about sales.
Here at AW... (esp the Share Your Work section) you will find the subgroup "mainstream/contemporary" which has nothing at all to do with sales and more to do with the general topic of the work. It implies that the work in question is not genre writing. It is listed in this manner so that if you are writing a book that is aimed toward general audiences, and it isn't genre writing, and does not have distinct modern literary tendencies and has a timeline which may...
or may not... be a little different than your own... there's a place for you to post. =)
Contemporary means that the author has set the book in the same time period as the one they live/lived in. The writer that wrote a fiction novel set in the 1950's while he/she was alive and writing in the 1950's... wrote a 'contemporary' novel. The author that writes a book now -today- set in the 1950's... is not writing a contemporary novel.
Literary does not 'just mean what it says' nor can you accurately or easily describe the category by reading the more famous literary writers. The many and varied characteristics of modern literary fiction can be applied to
any category of novel. You can find genre novels written in the manner of modern literary fiction.
If your book fits into any of these things, other than contemporary, which means nothing by itself, it will be surprising. None of these are genres. You really need to know what genre you're writing before you write page one, or chances are it won't fit anywhere.
It would NOT be surprising if a work of fiction fit into a category. It happens every day, and... no one ever needs to know what -category- of fiction they are writing before they write page one. Just like no one ever really needs to know a correct answer to a question before they make a complete arse of themselves...