"Unthinkable" can have several connotations
1.) unthinkable = outrageous, scandalous, transgressive. Prohibited by such a powerful taboo that even thinking about it seems unforgiveable.
These acts can be absolutely essential to the story or included for mere shock value. In both cases, the result can be worthwile reading. (I'm not the biggest fan of provokation for the sake of provokation, but I can see, how some people, myself included, occasionally just need to be provoked).
I think the freedom to explore these boundaries is an important part of literature, but so far I've rarely done it myself, because I generally prefer more subtle touches and still lack sufficient proficiency not to paint these things in lurid colours. (Which does not mean that painting in lurid colours can't be interesting and worth reading, it's just not my prefered colour pallette right now.)
Then again, what seems no big deal to me, might seem terribly transgressive to someone else, so there's always that.
2.) unthinkable = unmotivated, out of the blue, absurd. Things you would have genuinely never thought of, not because some sort of super-ego constrains you, but simply because they are so random and bizarre. Those are the things that leave us utterly bewildered. They defy conventional logic and resist any explanation. They make no sense at all in our usual state of mind (but might make perfect sense in a different one - see dream-logic/drug-logic/fairy-tale logic).
Many outrageous, transgressive, scandalous acts are really not unthinkable in the sense of unmotivated, random, absurd. On the contrary, the motivation is often rather transparent and banal. Take the Austrian man for instance, who imprisoned his own daughter in his cellar, raped her repeatedly over the course of decades and had several children with her: Unthinkable in the sense that I don't want to go anywhere near that mind, it makes me so sick. But then again, it's absolutely no mystery to me why he did it. There isn't anything random about it. It's nothing but the continuation of a vicious cycle of abuse (the man in question was regularly beaten by his mother as a child), taken to extremes by a logical escalation of a misogynistic belief system and a patriarchal sense of entitlement. Makes perfect sense, in a rather banal way.
Personally, I feel more drawn to the unthinkable in the sense of the Absurd. I'm not sure I can pull it off. The art lies in making it seem perfectly natural, while your reader is immersed in the story, and have them register the absurdity of it all only at the back of their mind, not in a way that takes them out of your world - just like your dreams makes perfect sense while you are still dreaming, even if you might somehow subconsciously know that it's all a dream that will fall completly apart once you are awake again. In order to do so, you have to convert your readers to a different state of mind, which is certainly no small feat, isn't it?
I think it's done perfectly well in Alice in Wonderland and many other stories directed primarily at children, as well as in anything written by Kafka.
I'd love to be able to do that but I find it extremly difficult. I'm unfortunately very invested in conventional logic and find it hard to let go.