Octavia Butler's Kindred is like this; it's about a modern (1970s, when the book was written) woman who time-travels to the antebellum South. While its depictions of plantation life and slave experiences are rigorous enough to satisfy any reader of historical fiction, I'm pretty sure the book is typically considered science fiction or historical fantasy, because of the time-travel element and because of what the rest of Butler's oeuvre looks like. And that's fair; whereas a historical novel about the period would be a story of life in the antebellum South, Butler's story is as much about that life as it is about how a modern person reacts to that life and interacts with the people who live it. It's quite a different focus.
I have to admit I read Kindred book because of its historical element, but I knew it had a time-travel element; I could imagine that an audience in the market for straight-up historicals might be put off by that aspect, because usually people who read historicals are looking for faithful representations of the period and are comfortable with speculation taken so far and no further.
Long and short, I agree with your conclusion to think of your story as time-travel-science-fiction with a historical element, rather than as historical fiction, because the former probably better captures your audience and what they are looking for.
