Not counting books that I tried one title of and never pursued...
I gave up on Dragonriders of Pern after All the Weyrs of Pern, as events in that seemed to betray the very basis of the Pern colony - IIRC, that's about when her son started becoming involved, and maybe when the author's mental issues were starting to catch up with her. (SPOILERS - She killed off Masterharper Robinton in a move she later admitted regretting, and she re-introduced high tech to a colony that was founded specifically to eschew high tech... while setting up the permanent end of Threadfall and thus the impending end of resource-hungry dragons, which were kinda the point of the series.)
Speaking of McCaffrey, I quit her Brain and Brawn ship universe when the third one adhered to the exact same plot formula as the second and first.
I quit OSC's Enderverse by the third book of his Ender's Shadow quartet. Too much of the author's politics were tainting the plot and bending the characters into mouthpieces.
I gave up on Eoin Colfer's Artemis Fowl books after the fourth installment; it worked better as a trilogy, and felt forced. If I were bored and found the next volumes at dirt-cheap prices, though, I might consider picking them up, but the TBR pile's awfully deep...
I stopped Susan Cooper's The Dark is Rising sequence after Book 2. Just not a fan of religious fantasy, especially highly orchestrated battles between Good and Evil.
Gave up on Margaret Peterson Haddix's dystopianShadow Children sequence; first book was nice and darker than I'd expected, but the second felt too manipulated, so I lost interest in.
Elizabeth Haydon's Symphony of Ages series started with great promise, but Book 2 became too much of an angstfest. Then they threw in deliberately wiping a character's memories to further the angst... gah.
I tired of Robin Hobb's Rain Wilds Chronicles after the overlong second volume... kinda got tired of Hobb's sprawling Farseer universe as a whole round about then, too.
James A. Owen's Chronicles of the Imaginarium Geographica started good, but I tired of it midway through the second; the premise of using real-world people in fantasies is tough to pull off, and the suspension of disbelief became a hard break.
I tried two books of Alexander McCall Smith's No. 1 Ladies' Detective series, but couldn't stay interested; I wanted more of the mysteries, less of the soap opera of people.
And, in graphic novels, I'm about one volume away from giving up on the Princeless series by James Whitley. Started as a great, fun subversion of tropes, but it's feeling really stretched.