I don't think women in general shy away from fiction with speculative elements. Many of the people in my fantasy writing groups, and the majority of people at the workshop I attended last summer, are women. And look at all the romances set in very fanciful historic or speculative environments.
I think there's just a human tendency to knock things we don't care for, and since many of us were taught (by our high school and college English teachers) that the only "worthy" reads are the classics, or modern works that deal with "realistic" scenarios, it's easy to roll one's eyes at another person's taste in genre fiction. I think people who like romances, spy thrillers and even mysteries get this too. People can be very dismissive in other peoples' taste in music too.
I would argue that the recent success of fantasy and its emergence into mainstream pop culture may be why some people curl their lip at it. There's a very human tendency to knock something down if you feel like everyone is suddenly into something you don't "get."
On a personal level, I have run into more fantasy fans (especially younger ones) in recent years, and they definitely don't all fit the old stereotypes of fantasy buffs. I was talking to a couple of my students the other day and writing came up. When they asked what sort of stories I write, I said fantasy, and they both said they loved those kinds of books. One was male and one was female. Actually, a huge number of the people I've met through fantasy writing groups and the workshop I attended last summer are women.
