Every book in a library has been purchased at some point (theoretically), and the copyright holder(s) paid once. That's how book sales work. A library is, in this sense, no different than a used book store. If you buy a book at a used book store, the copyright holders get nothing from that sale.
Piracy eliminates any profit to the copyright holder.
caw
This, and also, it's possible that the overwhelming majority of people who patronize libraries do buy books as well. I did even when I was younger and much poorer than I am now. It was more of a treat to go on a bookstore binge back then, but when I could, I did. I loved (and love) owning books--seeing the rows of titles on shelves. I've also been introduced to authors (and took a chance reading them) because I found one of their books in a library and decided to give it a try. If I liked it, I usually would go and buy other books by that same author that weren't in the library (or even were, but I liked the author well enough to want to own their books). Nothing like reading the first book in a series (or finding one somewhere in the middle) for free and getting hooked
I think the same principle applies with those freebie promotions on Amazon etc.
The point is that going to a library is not the same thing at all. Neither is borrowing or swapping paper books with friends (legal) or patronizing used books stores.
As for the whole "pirating an e-book is no different from sharing a book you've read with a friend" argument? Yes, loaning or giving away paper books is legal, but that comparison just doesn't wash. I can lend (or give) a paper book I've purchased to one friend at a time, but if I want to copy an e-book I can give one to every friend I've got
and keep my own copy. Heck, I can give them to people I barely know, or don't know at all on line. The number of lost sales is much greater.
The reason why "borrowing" e-books without paying for them is so problematic is that one isn't limited to just borrowing one at a time (a single copy can be used to make dozens, even thousands more), and one can do it for any book one wants. So the incentive to buy any books at all ever will be practically nil with a pirated e-book (or pirated music or movies or anything else). Liked the book? Just pirate more.
I know a few people who hardly ever buy new books (whether for financial reasons or some other reason) and mostly read them as loans from friends or via libraries, but I think they're the exception to the rule. Most people I know who love books, and who don't want to steal, love purchasing and owning copies when they can afford to.
Pirating is a terrible thing to do to authors. Most authors I've known love libraries, though.