Well, going far, far back into the origins of literary works, the original "romances" were serial (episodic) metrical poems of daring-do, of knights and ladies and courtly love affairs. But in the 1500's, romances began to include prose--brought about mainly by Spanish novels that involved heavily supernatural elements with love involved only as a secondary thread. But modern, 19th and 20th century "romances" evolved beyond the adventures to focus primarily on the love affair between the two main characters. That's not to say that every novel involving romance is necessarily SHELVED in romance. There are quite a few of what are deemed "love stories" that are shelved in General Fiction or Mainstream Fiction.
The difference between what is shelved in Romance versus the "love stories" are whether a) the relationship is the focus of the book, and b) the parties achieve a "happily ever after" (HEA.) Presently, novels that do NOT include an HEA wind up either in Women's Fiction (which are basically female-centric tales of life and living) or General Fiction. Historical fiction doesn't really have its own section of the bookstore, unless it's a Western (meaning United States West from 1830-1900), or shows up in the Regional section.
That's not to say that publishers aren't looking for historical, but the term "historical" precludes an alternate history--the goal is to fit the elements within KNOWN history. An alternate history literary novel with a strong romantic element would, as veinglory said, probably wind up in Fantasy. It's not a bad place to be, overall, but if your goal is to have the book treated as a serious literary work, the placement on the shelf will be difficult to overcome.
However, I wouldn't dismiss the romance shelves as somehow inadequate for your needs, since romance readers might be more discerning than you believe. Long gone are the days of trite, overblown stories. There are any number of current romance novels that are quite literary, and keep in mind that romance readers account for over 50% of the book buying public.
Or, is there more to your question than appears?
