I don't necessarily believe literary readers/writers are "self important", but I believe they look for more in the literature they peruse. Escapist stories (which I prefer) don't usually make the reader work for the nuances. They are usually not loaded with allegory, and the metaphors are all spelled out. Literary stories have to be read on a word-level (almost a syllabic level) to get all of the nuances, to make clear what's happening, and the metaphors are in the scene, not the sentence.
Granted, if written poorly, literary prose can be pretentious; but if well written...it's brilliant (anything that would lose a purely escapist reader would be so buried that they'd enjoy the story on a more primeval basis, and never notice what they are missing).
Any literature can be deep. I've read Fantasy and Science Fiction that forces the reader to consider what's possible, or what the true human condition is--and that's deep--without being inflated or pompous. It's the ability to bury hints that fulfill the discerning reader on multiple levels that makes a story "literary".
The only form of literary that grates on me (which is far more common than it should be) is "slice of life", where there is no plot, rhyme or reason.