I... don't think this has ever mattered to me as a reader.
Portal fantasies are popular.
Secondary worlds/universes that never mention or connect to Earth are popular.
Alternate realities - which sounds like what you're talking about, a galaxy in which there is an Earth but there is also interplanetary magic - are also popular. (It's also a Thing that Earth is implied but not named; I recently read a sci-fantasy in which humans were the only sapient species in a magic-powered galaxy, but Earth was not named, only references to a place called "Origin" that was likely lost to history.)
The important thing is not to force it: don't dance so far around Earth if there is an Earth in your invented galaxy that it tires the reader out keeping up with the steps, and don't force the existence of an Earth to explain humans if there is no Earth. (Note that George Lucas, to my knowledge, just had humans a thing in his galaxy far, far away - no convoluted explanation how a non-Milky Way galaxy parallel-evolved H. sapiens was needed, or whether they were "true" Earth-origin humans or simply humanlike analogues, 'cause it was a story about space fights and light sabers and the Force, not about how each sapient species came to exist and invent space travel - and the audience understood that humans were writing and filming and acting the story for a human audience anyway, so outside of a handful of nitpickers nobody really cared.)
What works for your story? Do that.
