Trade publishers only have room to publish so many novels. They look for the most marketable ones to acquire and publish. Agents know this. They look for manuscripts they think are the most marketable.
If you (generic you, not necessarily the original poster you) are looking to trade publish, there are two routes to take. A) Write the book you want to write, then see if it fits. B) Write a book you think fits the genre expectations of your preferred genre.
Option b means reading widely in your genre and research into what is currently selling. Option a is a risk, but usually means a book you are passionate about getting out there. Keep in mind that a writer's first book might not be their trade published debut novel. Just finishing that first book is an accomplishment and (hopefully) makes the second, third, fourth book easier to finish. And keep in mind that the most common mistake new writers make is querying a manuscript that is not quite ready yet. With that much competition, it needs to be as good as you can make it.
So, there is a risk that any manuscript won't be accepted by an agent or trade publishing house. A few trade publishers do take direct submissions, most only take agent submissions. If you are lucky enough to get trade published, well, many books don't pay out their advance, let alone earn royalties. My understanding is that would be a loss for the publisher. Definitely read and understand your contract before signing any deal. Which is as much legal advice as I'm willing to give with AW's no legal advice policy.
As for advances, not all trade publishers are large enough to pay advances. Others might pay in the four figure range. How low are you willing to accept to get your book published? Keep in mind that publishers are good at knowing how much they can make from a book and their advances reflect that knowledge. Depending on how successful the publisher thinks your manuscript will be or how good your agent is, you might get as much as a six figure advance. But as I said above, any trade deal is possible but unlikely. If you can point to successful trade published books in your genre/format, it is a bit more likely than if you can't use those comps in your query letter.
###
A vanity or hybrid publisher will charge you (again, generic you) to publish your book. You will likely have to market it yourself. If not, you will again need to pay the publisher to market your book. It can be done profitably, but it is difficult unless you know what you are doing. If any publisher asks you for money upfront, they are not a trade publisher, they are a vanity publisher.
###
Self publishing is easier than it has ever been. Technically, vanity publishing could be considered part of self publishing since in self publishing you (again, generic you) are doing or hiring everything yourself. It has more of a learning curve, since you are doing everything from cover art/design to editing to marketing once you toss that ebook up on a selling platoform or arrange PoD books from a printer. There can be less of an initial investment than a vanity press package or more if you are hiring everyone independently. If you do everything yourself, there will be minimal upfront costs, other than time. If you hire most of what needs doing, it might be worth going with a vanity press's package with the services you need rather than hiring multiple individuals for each step you need help with.
Cover art alone, from what I remember elsewhere on AW can range from a token amount to thousands depending on a variety of factors. If you use public domain clip art, the final composition might be copyrighted (you'd need to ask a lawyer who specializes) or it might not. Another common cover shortcut (see
Critical Theory threads for more on the subject) cannot be copyrighted. There are members who successfully self publish.
###
I had to pick an order. There are trade published members on AW. There are self published members on AW. There are unpublished members on AW. I went with an order that made sense to me with vanity publishing between trade publishing and self publishing. It is not my intention to imply any one publishing method is better than another method.
