When I had a self-pub project, I read five or six solid books on typography and layout (I had searched through the self-pub forum for title recommendations!), and then used Scribus to put together my layout. My printing was done through Lightning Source.
I probably had an easier time with Scribus because I'd already had a little experience with helping a friend put a catalog together for his biz using Microsoft Publisher. So I had a rough idea as to how desktop publishing software worked (versus, say, a word processor), what to watch out for, and things like that. And likewise, because of other experiences in my background, I had a better eye for font readability, how to appropriately mix fonts, tracking/leading/kerning, and stuff like that.
I'd definitely recommend Scribus--- but I'd recommend checking out some good books on book design first, so that you have a direction you want to work towards. We're so used to seeing good book design (in terms of typography and layout), we take it for granted, and it can be invisible--- and when we run into bad book design, it's not always obvious as to what's the matter. Having the research under your belt helps you know, "Ah, this is a modern typeface with a vertical stress and thin, flat horizontal serifs. It's going to be hard on the eyes if I use it in a block of text, so I want a serif typeface with a diagonal stress and less dramatic emphasis between the thicks and thins."
