The also sad fact is -- too many people equate rape with violent, forceful sexual activity that leads to hospital stays, lasting physical wounds, or death. That's what they see on TV and in movies. They hear supposed experts differentiating between "real" rape, and the kind that isn't physically violent - the kind they refuse to label as rape. So when someone's drunk and seems to be in on the fun, or doesn't put up more than a seemingly token defense, or doesn't scream and shout and kick, then that can't possibly be rape. They see someone "willing," when in fact, the person is inebriated and not legally able to consent.
I think this is definitely a big problem.
And I think some people have a hard time accepting things like date rape and marital rape because a lot of times, the perpetrators are "nice" people. They're not disturbed strangers who lurk in bushes--they're people's friends and family members. Even the victims might think they're nice prior to the attack.
I think this is one reason why victim blaming occurs more with sex crimes than with crimes like mugging or burglary. People who commit burglary or armed robbery are probably unlikely to be respected pillars of their communities, or kindly middle-aged family men. But rapists are in all walks of life. Some people have a hard time accepting that.
I also think that some people have a sort of warped perspective about what lack of consent really means, and how much context can matter. There can be this defensive attitude that rape accusations are something that happens to a man because he's not careful or perfect enough, even though in reality, that's pretty unlikely.
For example, I got into a debate once with this guy who had a really big problem with the idea that men should stop immediately if a woman asks them to, because it's physically impossible to stop having sex instantaneously. He was fixated on this idea that a man could be accused of rape just because it took him a half second to fully remove his penis from a woman's body.
Well, yeah, technically speaking, it's impossible to do something instantaneously. But this misses the point entirely. Women aren't saying "No" and then whipping out a stopwatch to see how many milliseconds it takes for the guy to pull out. That's ridiculous. I think most people can tell the difference between a partner who's considerate and someone who doesn't care/is ignoring their requests to stop.
But there can be this defensiveness, and this idea that rape accusations "just happen" no matter what you do, or that women have all this power over men.