With classic forms of SF and fantasy, it's simply the setting. Space ships, alien planets, and the future=SF. Quasi-historical settings, dragons, magic-fantasy. But both genres have long since expanded beyond these forms.
I think it's often as much about the premise behind the setting setting as much as the plausibility of the made-up elements. Dune is considered to be SF because its setting is the the space-faring far future. Same with the Dragonriders of Pern. Same with Star Wars. I suppose you could call them soft SF, or fantastical SF or some such thing. Yes, all of these have mental powers and technology that are magical that allow them to distort the known laws of physics. But the explanations are quasi scientific (psi powers, or the spice mutates the navigators so they can warp space and time with their brains). And the places where these stories occur are a part of our (aka the real) universe and is supposed to be operating by the same rules, even if they're fudged.
Even Star Wars is set in a galaxy long, long ago and far, far away. Still a part of our universe, though very distant.
The Wheel of time, Lord of the Rings, and George RR Martin's books are fantasy, because they're set in either some long-ago past (or future with WoT) where the laws of nature were different, or some alternative world/universe we could never reach with technology alone. The fantasy universe is has its own rules (often explained by magic, or via gods that don't exist in our universe).
But there's this huge area of overlap when it comes to historical fantasy (versus SF set in the past), steampunk, SF and fantasy set in the present-day world or near future and so on. With these, I think it comes down to whether or not the things they have that we don't (or never did in the case of historical settings) is whether we invoke magic, a change in the laws of nature, or the supernatural to explain it or not.
So if you write a novel where some people discover they can use mindspeech, and it turns out to be because they have the blood of an ancient race that was touched by the old gods flowing through their veins, or maybe even just a cool and unexplained magical power? Fantasy. If it's the result of special psi powers because they're the results of secret genetics experiments that created brains that could interface with subatomic particles in a way that opens communication channels between people? It's SF.
Just my two cents, but of course it's the nature of the beast that we'll all be able to find works that lie in the blurry boundaries or that incorporate elements of both.