My parents were sf fans and hung out with sf authors. It just seemed natural to read it.
Historically, many stories have had fantastic elements in them. The realistic slice-of-life "mainstream" stories (or genres which could take place in the real world, like war novels and westerns and romances) are a relatively recent development (in the West. Not counting China or Japan here, although perhaps I should).
Thing is, people have always enjoyed stories with imaginary elements in them.
These days just about every piece of fiction which is not strict realism is classified as science fiction or fantasy (I consider horror a part of that, but I'm willing to entertain it as a separate genre). It's not surprising that stories whose only common element is they can't happen in the world as it is would be a vast category containing many subcategories, and that it would have a very large number of readers.
Coming from my profession, it's as though there were two categories of painting: "realism" which is considered mainstream (and contains photorealism, Victorian realism, American regionalism, etc.), and everything else, called "fantasy" 'cause it's not realistic and encompassing symbolism, cubism, surrealism, Impressionism, medieval manuscript illumination, depictions of anything imaginary, mythology, speculative science illustration, allegory, and on and on -- anything, anything not strict realism.