I think the fact that romance and erotica both are based in human fantasy is what makes drawing the line on this so difficult. For instance, one of the biggest fantasies among women is the "rape" fantasy (one could actually argue these are domination fantasies, not rape), and yet, it's also the most abhorrent thing in real life. But these fantasies work because even the "unknown attacker" is known. They have limits, their desires are always the same as the fantasizer, and even their looks are acceptable. In the fantasy, even at their most dangerous they're a neutered attacker, completely safe, and the outcome will always be what the fantasizer desires.
Erotica and romance have the same traits. These are shared fantasies--as someone else said, the suspension systems will never fail in a fantasy. On the same note, duels to the death won't leave the hero crippled, rough sex will cause no damage, and if someone is being an ass, they'll always wake up to it before the emotional damage become permanent. Oh yeah, and STDs only exist except conceptually, while unwanted pregnancies are a mere plot point with any hardships either overcome or completely forgotten, even by the people around them. It's the world with the rough edges removed.
So I do allow a lot of leeway for unlikely behavior that wouldn't fly in the real world, because, to me, that is the nature of these sorts of books. They're an escape from reality, not an embracing of it, a safe walk on the wild side.
That said....
A pet peeve of mine is people portraying dangerous or unlikely behavior and/or equipment in BDSM, but for me it's because I like well researched stories. If the author is unwilling to research their subject, I won't respect them. I feel like authors have to earn wiggle room to stretch boundaries (the mind/emotion reading dom who never needs a safeword, for example, because he knows what you want before you do) by first getting the foundation right.
Where I truly balk and it slides into abusive for me is characterization.
Back in the big brouhaha surrounding Twilight, I read an article that had the most fascinating point to make. It said that Edward and Bella's relationship--creepy points and all--might have been more acceptable if, instead of portraying it as a great, sweeping romance that was good for everyone, it had been portrayed as a relationship that worked for them. This shifted some gears in my head because it was so right. I had read about, cheered for, and fallen a little in love with a lot of heroes and heroines over the years, including ones doing things in their relationship I would never want in mine, or people I wouldn't be compatible with or even like in real life. I didn't cheer because they met my ideals, but because they were so right for each other.
And many of my wallbangers were the places an author tried to force two incompatible characters to hook up and make me believe it was happy and would last. Hermione and Ron are a sterling example--those two have marriage counseling and possibly a divorce in their future, and frankly I'm none too sure about Harry and Ginnie's longevity either considering how he constantly ignored what she wanted and left her behind. (Yeah, a silly example, but one everyone knows.)
Character is everything. So is showing romance in the microcosm of two people, not the macrocosm of "this is normal to everybody." Because no, it never is.
The one that always comes to mind was a romance I read a couple years back. In it, the heroine did something wrong; sent a letter telling the hero the women he liked (the other woman, because he also liked the heroine and frankly liked her more) was getting married to someone else. Things to keep in mind is even the hero knew he'd never get to marry that girl, he openly admitted they were unsuited as a couple, and the girl was actually marrying someone else anyway, though that wasn't yet known to the main characters.
Oh, and the hero discovered the deception before he married the heroine.
His reaction was not to call off the wedding.
Nope, his first reaction was to rape her until she broke, body and mind. No, that isn't hyperbole on my part; in the most lyrical, beautiful prose, he thought about how he wanted to physically and mentally wreck her for the travesty of...sending a letter. Of being so desperately in love with him (and a teen at the time) that she pulled a boneheaded stunt. Then he changes his mind, deciding it would be better to marry her and emotionally abuse her the rest of their lives. Which he proceeds to do for the next ten years. And when she finds a better guy and demands a divorce he is once again ready to harm her, because how dare she try to leave?
And the focus of the book? The change arc was about the heroine realizing how wrong sending that letter had been because she had (and this was openly stated) wounded his manly pride. Once she finally owned up to that, everything was okay. The hero had no change arc, as everything he'd done was apparently perfectly acceptable, even romantic. I mean, what else do you expect from a man?
Those kinds of portrayals of abusive relationships scare the bejeesus out of me because they package abnormal and even criminal behavior as something to be expected of men, as base normal for the gender. They teach men really are monsters at heart, and changing those monsters means bending to them until they stop hurting you, and loving them until they see you're worth loving back.
All of which never works in real life.
I am genuinely okay when I can see and know it's just a fantasy between two decent, or even morally grey characters, when the author shows me not "this is a functional relationship" but "this relationship works for these two people in particular because their deficiencies, demons,and good points all mesh into a positive whole." It's when the message comes that whatever bad behavior they're displaying is normal, expected, and common to everyone (and romantic to boot) that I find myself once again denting a wall with a well flung book.