I think SF is inherently naturalistic. Arthur C Clarke said 'any sufficiently advanced technology is indistinguishable from magic', but SF still believes there's a distinction. The more you rationalise magic, the more SF-like your story becomes, I think.
LOTR doesn't rationalise magic in the slightest. We don't really know what magic is capable of in Middle-Earth; we have only a very sketchy idea of what Gandalf can do. It's a surprise when he comes back from the dead, but there's also a sense of wonder about it. Merlin, similarly: we don't know what he can do, what the limits of his magic are, and it's used sparingly.
If you rationalise a bit more, you can come up with something like Ursula LeGuin's Earthsea, or Patrick Rothfuss's books, or Harry Potter, even. Magic in these books has rules, and formulas, and is studied like a science. It seems the more you use magic in a fantasy novel, the more it has to be explained and codified - you have to start accounting for the fact that a wizard is imperilled, or can't escape from a prison by walking through walls.
I'd also mention Randall Garrett's Lord Darcy stories, or Jim Butcher's Harry Dresden series - essentially mysteries, but incorporating a lot of carefully-explained magical lore. The rules are made very clear, so that Garrett's wizardly forensic scientist can use voodoo to match a bullet to a gun without offending the sensibilities of detective fiction, for instance. Dresden, likewise, is basically a fantasy Spenser, PI.
I think they fail to shade into SF because however rule-based and heavily-explained their systems of magic, they don't try to explain magic in naturalistic terms. We don't get a situation where Harry Dresden throwing a fireball is explained as a mutant organ that allows him to tap into zero-point energy and send a negative charge through his flux capacitor. The characters, and the narrator, are content to explain magic in metaphysical terms - force of will, that sort of thing - and not look into it with a microscope.
Obi-Wan Kenobi talking about the Force in Star Wars is just about on the border, which was one of the things that made the first three films interesting - the mix of Han Solo's gritty materialism, hydrospanners and all, and mystical, elusive magic practised by warrior priests. Then of course Lucas rampaged wantonly over that border in the sequels with all that Midichlorian bullshit, took a wrong turn, and ended up in the barren wastes of Crap SF. The first trilogy had archetypes as characters; the second had clichés.
Anyway. Your story needs internal consistency. In a highly rationalised fantasy, or a piece of SF, you will have many rules about how magic or technology works. You can only cast three spells per day; the Enterprise cannot exceed Warp 10. But in a lightly-rationalised fantasy, magic is something whose use just has to be consistent with effective storytelling. We don't really need to know precisely how the One Ring works, in physical terms, or how Galadriel does her thing. Moments of magical intervention in a Tolkien story are almost religious in nature; miracles, in fact. As miracles, they're justified in a story by the story's shape, by some sense of justice or catharsis.
And of course that applies to the other end of the scale, too - if it's dramatically necessary, and exciting, and thrilling for the Enterprise to reach Warp 11, well, that's what we're going to do. Later someone will write a bit about how they reversed the phaser bank buffer through the warp coils. (It doesn't really matter; Star Trek's audience will forgive a certain amount of hand-waving, and like most fandoms rather relish figuring out ingenious solutions to rationalise it.)
Where you're not going to get away with any polarity-reversing is in hard SF, or god help us Mundane SF. Right. I've gone on long enough. Off for some dinner.
) and I don't know any different, I'll believe them. But maybe other readers are more picky than me???
