I would say, fantasy is based primarily on what we consider to be mythology (often but not neccessarily European mythology), whereas science fiction is based on speculation about what we consider to be the "real" world. Fantasy is also often based on folkloric or theological themes such as quests to find magical objects or to lift curses, lost princes, redemption from evil and etc.. While these themes also occur in SF, and indeed in all fiction genres, they are usually covert - whereas in fantasy they are overt and explicit.
Very "hard" SF tends to be ideas-driven, where the science-element - the new technology or whatever - is the main focus of the story, whereas fantasy is usually plot and character driven: but a great deal of slightly less "hard" SF is equally character-driven. I think for example of CJ Cherryth's Chanur saga (my own favourite SF series) which is mainly about politics and war and commerce and detailed character-development and even romantic love, but all set among alien societies and against a high-tech. background which are both absolutely vital to the plot, and not just a technological gloss on a fantasy story.
There is inevitably some overlap. SFEley referred to "concepts...with no scientific grounding, like telepathy" but in fact telepathy, although still unproven, is a subject of hot scientific research, and is a rather more likely and less fantastic element for an SF story than, say, anti-gravity.
Mistri mentioned the Pern books, which are solid SF which read like fantasy, with their genetically engineered dragons etc.. Tad Williams is another interesting borderline case because he is a fantasy writer who reads like SF.
His The War of the Flowers revolves around a high-tech. fairyland, and if it wasn't for the fact that some of the characters are little fluttery things with wings (and horrble bad attitudes) it would be a straight-SF parallel world story. The Memory, Sorrow and Thorn series is said to be based on Tolkein's Middle Earth, a thousand or more years later and on a different continent: but while Tolkein's elves are mythological and quasi-religious, Williams' version feel like some sort of alien life-form.
Idealization may be part of the key, and part of the distinction. Tolkein's fantasy elves are idealized and are there to represent certain qualities, to be beautiful and mysterious and, yes, fantastic. Williams' nearly-SF equivalent are treated as a well-rounded species with their own faults and virtues and individual natures, and are as much weird and disturbing as they are attractive.
I don't care much for fantasy of the dragons and quests type, but I am currently reading Barbra Hambley's Darwath trilogy, and she is another author who overlaps the boundaries. She writes about magic and wizards which says "fantasy," but does so in the context of a well-thought-out system of parallel worlds, in which magic is portrayed as just another sort of energy which operates more strongly in some universes than others. Her inimical Dark Ones read far more like alien energy-beings than fantasy demons.
I have a potential problem with the distiction between SF and fantasy myself, because I'm writing a novel which I would consider to be SF - albeit SF in which the sciences concerned would be mainly sociology and evolutionary biology, rather than technology and physics - but which may end up being classed as fantasy, simply because the setting is an alien world where the technology-level is quasi-mediaeval and clairvoyance is an established fact.