Literary: What is it? V.1.1
Any story or novel, no matter what the genre, can be "literary," where "literary" means written to the highest prose standards.
What people really want to know, most of the time, is what do those blasted agents and editors mean by "literary?" Here's my take on it.
When a market says it's looking for the literary, I assume it wants fiction set in the present or the recent past, emphasizing character rather than plot, more static than kinetic, and displaying an attention to language and style that can verge on the poetic. Sometimes markets call such fiction "contemporary." If it deviates at all from the depiction of "real life," they may call it "magical realism" or "surreal." If the structure or language are noticeably funky, it's "experimental." In addition, there are a lot of other descriptors that I associate with a desire for literary submissions: ethnic, feminist, regional, multicultural, erotica, gay, historical, prose poem, and so forth. (NB: This doesn't mean that no genre fiction can be ethnic, regional, gay, etc. These are just catchwords I constantly find in the lists of acceptable material put out by mags, journals, publishers, and agents.)
As for mainstream (that word so often paired with literary), the best definition I can come up with is: Literary, but not so much so, maybe something actually happens, could even toy with subject matter associated with the milder genres, like mystery and romance and suspense and thriller.
When the market doesn't want genre fiction, it doesn't want: Mystery, romance, suspense, thriller, adventure, or western. These are what I call the mild genres. Some markets would also throw religious and historical into the mild genre category. Some throw in children's and YA as "genre," but I'd rather call them "intended audiences."
And now we come to the Three Deadly Genres, which are so evil they are often mentioned by name in the lists of what-we-don't-want: Fantasy, Horror, Science Fiction. Oooh, my favorites. I must be going to Literary Hell on a particularly fast handcart.