Self-publishing has earned me the most money hands down. I write interracial romance (BWWM) and that type of romance was born from self-publishing and ebooks and never was a genre that publishers focused on or acquired. Especially not the IR books with black heroines. I used to be with one of the big five for another genre and I made more my first year self-publishing than I had the four years I was with the big five press including more then the advance I got. I still get tiny royalties from the big five house though.
It isn't all about money for me though. I enjoy the process of self-publishing, and I write first because I love it. I'd write even if I never made another dime. Of course I am happy whenever I sell something though.
A lot of things go into what might be better for an author and it has nothing to do with the book itself. It's about the type of writer you are, your goals, and expectations. For one thing, self-publishers tend to release books much faster than publishers. You'll find many self-publishers that release way more than 2-3 books a year. I definitely do. Some self-publishers release that many in a month!!! So how much you release has a factor on your income too in self-publishing.
Everyone's experience is different. Also each book is different. You can't say what a book will or won't do even if it's in a popular genre. We don't control the market. Whether or not trade or self-publishing would work is up to what the person writes, their expectations, and goals.
An important thing about self-publishing is you gotta love it because it's NOT easy. It's a lot of work because everything is on you and you only. If you aren't the type who wants to be in charge of the entire process, the good and the bad, then self-publishing is not for you. The ones who make self-publishing work for them are those who remember it's a serious business and you're no longer just an author when you step in those shoes. You gotta
want to self-publish because if you go into it thinking it's easy or thinking it's a quick fix because you are having a hard time getting a publisher, you won't make it. It's not to be thought of as an "out" for an author having a hard time elsewhere. It's a legitimate form of publishing and should be done with respect and taken seriously. Serious self-publishers take pride in our work and do our best to release quality books.
No matter what mode of publication a writer is interested in, it takes dedication and hard work and there aren't ever any guarantees even if you do everything right.
The key is choosing what's right for you and sometimes doing both might be.