Just back away slowly, smiling in a vague manner.
When I was a kid, some family members held a garage sale and invited a bunch of people in the family to come sell stuff. Some of my family tend toward having a lot of books, and thus we'd set up some bookshelves on the porch (in case of rain - most other stuff was out in the open because it was ok to be rained on or easier to move quickly) filled with books for sale, all nicely segregated by category and such.
A group of four adults and two kids (I can still picture these people and they seemed small kids, their parents and a set of grandparents) were shopping and had chosen some items. One of the adults asked if there was other stuff and more kid stuff and one of my relatives replied, 'oh, we've got a bunch of books over there; there are kids' books and a lot of mysteries and...' and was cut off when the middle-aged man said, completely scornfully, as if it'd been suggested they go play in poo, 'we don't
read.' The grandparents both scoffed/laughed and shook their heads disdainfully, like they couldn't believe we'd suggest such a pathetic thing.
They put the stuff they'd been carrying down on the table in front of them and walked off shaking their heads.
We spent most of the rest of the weekend making up fanciful stories about them. They remain a family joke to this day. "We don't
read."
*Before this is taken the wrong way, plenty of people weren't interested in the books, or said they weren't interested in the books. No big deal. It was treating it as if suggesting reading was akin to 'oh, our fetish porn-making props are over there.' Not that there's anything wrong with that either, among consenting adults - you know what I mean!
In general, I think it's weird, but in the same way I don't get people who hate other stuff that I decidedly don't.