Here's how I see it:
Marketing is about identifying the customer and pitching a product to them. It's a very broad term and includes lots of activities, most of which are focused on connecting to a customer and directly trying to interest them in your product.
So in the University where I work, the Marketing department works with advertising agencies to "brand" the Uni, come up with posters and visuals and produce prospectus for different departments. Their focus is outwards to prospective students/parents of students; their aim is to try and interest them in attending the University.
For a book, marketing is directed towards readers (selling books to bookstores would probably fall under the remit of Sales - closely linked to Marketing in many cases). Marketing identifies readers who might be interested in the book ("Fans of <author> will love <new author>'s debut novel,
I'll Never Be An Emo Vampire!") and tries to sell it to them directly (posters, blurbs, events, release 'buzz')
Publicity is the public discussion of a product. Where I work in the Press Office of the University, our job is to generate publicity. We do this by writing press releases and pitching stories to the media about what the Uni is doing. When the Uni is mention in public (i.e. the news/on a TV show/etc) and that message doesn't come directly from the Uni itself (as with Marketing) then it's publicity and it benefits us and our reputation. Having your product discussed by third parties without your (apparent) direction is generally considered good exposure.
For a book, generating publicity would involve trying to interest the press in running stories or features about it (shopping the ARCs out for book reviews, giving interviews with the author, having signing events and inviting journalists). You are looking for any mention of the book
by someone not the publisher or author - a third party endorsement of the product.
Publicity is a double edged sword, especially in the internet age - it's very hard to control what the general public is saying about your product. You can't keep them "on message" as you would with a Marketing department. When you send the book out for reviews, there's always the chance it will be utterly slated.
Self-Promotion is the work you do yourself to market or publicise your work. You might self-market by directly advertising your book, or self-publicise by ringing round media outlets offering to give interviews.
In the context of the book discussion, self-promotion would just be anything the author takes upon themselves to do in order to further sales of their book. Self publishers obviously will have to do all their own marketing, and try to generate publicity for themselves, but any of their efforts could be classed as "self-promoting" because they do it all themselves. For an author with a professional publisher, much of the promotional work is done by the marketing and press departments of the publishing house - they will have to do very little self-promo work, probably talking about the book on their blog or mentioning it in tweets or whatever.
Obviously the exact amount of self-promotion an author is expected to do will vary depending on their particular circumstances and arrangement with the publisher. It's probably more likely they'll be asked to participate in the marketing strategy being run by the marketing department rather than go it alone on the internetz. But this will depend on the situation.
Just my take on the matter. I'm sure others can add/correct where they see fit.
