I think literary fiction can mean either of two things:
- Fiction that's thoughtful, deep, well-constructed and lasts; or
- Fiction that's experimental, self-indulgent and hard to enjoy.
Fantasy has a bit of both, but I think it's more productive to ignore 2) and just talk about 1).
There's nothing stopping fantasy or any other genre from being literary, and there's nothing stopping a story of literary construction from being popular among fans, so saying 'literary' vs 'popular' or 'literary' vs 'page turner' creates a false dichotomy. Editors like Sol Stein will often compare 'literary' to 'transient' fiction... the difference being that literary stuff is likely to still be read in thirty or sixty years time while transient stuff is likely to be forgotten. I like that comparison.
So, what's the difference?
In my opinion, literary stories will do more than just cash in on a faddish subject. They'll have well-constructed story that will speak across multiple generations. They'll have something thoughtful to say that other fiction isn't saying, and they'll say it entertainingly, evocatively and interestingly.
So, how does one do that in a genre like fantasy?
A common piece of advice is to make the story 'character-driven' rather than 'plot-driven', but I think that this is another false dichotomy and it's misleading to boot. Nearly all the romance genre is full of 'character-driven' drama (i.e. drama that features characters making choices and reacting to their inner needs), and most romance genre is transient fiction. Fantasy too has some quite forgettable 'character-driven' drama -- it generally comes in mauve covers with titles in curly metallic lettering and features young girls with cloaks on the front.
What I think makes a story 'literary' is actually balance and connectivity between character-development and plot. Literary stories have a rich palette of characters; the characters have depth and complexity; they contrast in interesting ways, and the plot emerges from
who they are, rather than simply
what they must do. Also they have strong themes and clearly focused concerns.
If you're looking for literary fantasy look for: stories with characters that break the mold for their roles; characters with credible and surprising internal contradictions; contradictions that emerge over time in response to circumstance; conflicts that are layered rather than simple; a character's masks of delusion and deceit stripped away over time; plot that never shoe-horns itself onto the characters, but rather emerges organically from the characters' own nature; and language that captures place and character and events vividly, memorably and concisely. Also look for stories that tackle moral concerns and savage them to death.
As two outstanding examples I'd nominate Ursula K. Le Guin's
A Wizard of Earthsea, and Gene Wolfe's
Shadow of a Torturer. Brief explanations follow...
In A Wizard of Earthsea (1968), Sparrowhawk is a talented magician who's too proud and ambitious for his own good. Rather than being a Child of Omen who'll Save the World, he screws up big-time and inflicts some Bad upon the world. Thereafter the rest of the book is about him working out how to deal with the Bad. By its own nature, the Bad is able to anticipate everything he does, and is never less powerful than he is so the better and more powerful he gets, the worse the Bad gets, so the battle is far more spiritual than physical. Sparrowhawk himself is a mass of contradictions: he's of humble birth but very proud; compassionate but sometimes ruthless; powerful but helpless against his own nemesis. His friends are well-differentiated, but often unable to help him. He seldom gets Items of Power to assist him in his quest. What he gets along the way instead -- often at high personal cost -- are Insights.
In Shadow of the Torturer (1980), Severian is an orphan who works in the torturer's guild. In a fit of compassion and adolescent love he shows a draconian sort of kindness to one of the guild's 'clients', and is thereby shamed and exiled. The rest of the book consists of Severian wandering the world, plying his trade on behalf of local governments and reflecting on what it means to be a guy who does a job that nobody wants to do, but which must still get done. Like Sparrowhawk, Severian is a mass of contradictions. He's sensitive and compassionate, but has the cruellest job in the world. He's deliberately measured in the cruelty he inflicts on others professionally, but he's sometimes far more cruel in his personal relationships. He has a very high sense of duty and honour, but he's constantly doing things for which he's later ashamed. He's self-sacrificing and has a strong sense of justice, but finds himself travelling with and working with some of the most selfish and unjust people imaginable. As with Le Guin's story, Severian's story is inextricably bound with who he is; if you put a different character in his place, it couldn't have the same experiences.
Contrary to the way it's sometimes touted, literary fiction isn't a litmus test for who's the smarter or deeper reader. Just as with any other fiction, one can enjoy one literary story and not enjoy another, and enjoying literary fantasy doesn't mean that one won't enjoy lighter stuff too.
But both these stories have deep themes, carefully developed, and to my mind all literary fiction does. In
A Wizard of Earthsea, a dominant concern is the exploration of where the wisdom of wizards comes from. So we see passages like this:
Ged, listen to me now. Have you never thought how danger must surround power as shadow does light?
From that time forth he believed that the wise man is one who never sets himself apart from other living things, whether they have speech or not, and in later years he strove to learn what can be learned, in silence, from the eyes of animals, the flight of birds, the great slow gestures of trees.
In
Shadow of the Torturer, a dominant concern is what duty a man holds to the state, and to his fellows. And we see quotes like this:
That we are capable only of being what we are remains our unforgivable sin.
Quotations like this make us stop and think. And getting the reader to believe them means that not everything that happens in a story will be pleasant and enjoyable, which leads me to some final points of difference...
Transient fantasy seeks to entertain us with its whimsy. Literary fantasy does that too, but also uses its whimsy to challenge and provoke us. Transient fantasy often features fear, joy, humour and triumph, but literary fantasy is most often bittersweet; it contains all the emotions of transient fantasy, but often its lingering aftertaste is a wise kind of sadness. As a general rule, fantasy that makes us feel too happy is seldom literary.
Hope that helps.