When I think about self-publishing, the main disadvantage isn't the stigma. It's the lack of distribution, the challenge of finding readers. The steepness of that challenge seems to depend heavily on your pre-existing readership (if any), your genre/category, and the ease with which you navigate e-book marketing. I don't think I understood just how much a publisher's sales/distribution network matters until I started seeing pictures of my book on shelves, all over the U.S. and way beyond. If I were writing in a category that sells like hotcakes in digital format, though, having books on shelves wouldn't be such a concern.
In my experience, many readers don't know the names of the smaller presses or even of the smaller Big Five imprints, so it's not easy for them to spot self-published books, much less apply a stigma to them. They go by contextual clues like "Is this book on a store shelf?" Many a time I have brought a bunch of review copies into the office, and colleagues have asked about each one, "Is it self-published?" They know it's a common thing, but can't recognize it.
OTOH, if you do something like put your cover blurb in all caps, there is a good chance people will guess the book is self-published, and the design flaws will reinforce their negative preconceptions (if they have any). If the book is well designed and well edited, they might never guess, or they might revise their preconceptions.
Me, I've seen so many books, including very accomplished self-published ones by hybrid authors, that I keep an open mind. I will say that I don't see a big quality difference between the average self-published book and the average very-small-press book. In both cases, the reader can usually figure out why this book didn't get a better deal, but may still enjoy the book. Both categories might have great editing, or (seemingly) none. Both have their outliers, good and bad. But the very worst books are on the self-published/vanity side.