1. Some readers like completely different types of fiction, and will see you not writing that type of fiction as proof that you are a bad writer.
2. Some readers will have built up their personality around an agenda, and will see you not writing a book championing that agenda as proof that you are a bad person.
3. Some readers will be "if only I had the time to write" non-writers*, who will read every sentence of yours through an "I could have written this better" brain filter.
4. Some readers will be people who can only feel special and smart if they tear someone else down, be it actors, musicians, scientists, sport stars, politicians, or writers--in this case--you.
One can either learn to deal with it, or one can say "screw dealing with shit I don't want to deal with."
Thomas Harris, Thomas Pynchon, Dean Koontz--they don't meet readers, don't do signings, don't maintain active online presences. Same way plenty of musicians and bands exist only as "studio projects", without live shows, without touring. No one is saying Darkthrone can only be a "real band" if they start performing live, and no one is saying Harris can only be a "real writer" if he interacts with his readers.
You do what you want to do and that's it.
___
* (or "if only I had luck and connections," embittered unpublished/unsuccessful writers)