Literary Speculative Fiction is my bread and butter. Literally. I went to the store today and I bought bread and butter (and sherry - lots of sherry) with money I earned writing exactly stuff under this definition
I like Strange Horizons. I also think you probably did read literary sci-fi in high school, because you probably read the original "Frankenstein".
Science-fiction, fantasy, and horror are all speculative fiction. I've never seen anyone throw mysery in, but urban fantasy like Jim Butcher is definitely speculative fiction, as is fantasy romance about werewolves like cathy clamp's book is spec fic, too. We needed a new term to cover such authors because elements of sci-fi and fantasy have spread like some kind of virus (or magic mushroom spores, if you prefer fantasy) all over the bookstore.
Different people will give you different answers to what this term means, like anything so subjective. However, for my money, the definition of literary is "ordinary people doing ordinary things". That's a quote from someone whom I've forgotten, but it's a good definition. Even with literary speculative fiction, there's a way to do it ordinary. (You have a couple of kids growing up on a space station. You focus on the mundane aspects of their life and eventual love affair in the shadow of a war, instead of the rocketship battles with aliens and zero-g sex scenes.)
Anyway, different people will give you different answers. that's just mine.