Hi Remus,
I remember you! I have been lurking on these boards since at least 2002 and but only recently started posting now and then. I remember you talking about your journey through the years. I'm sorry you have found things so difficult. Forgive me if you already mentioned this but what types of books do you write? Many times it's not the writing that's the issue, it's the story or genre. So your writing can be fantastic but if agents or pubs feel that the audience is too small they won't want it no matter how well-written it is.
Good writing is only part of it. The rest is the storytelling and the genre that's important. If the story sucks (not saying yours does) then they won't want it. They would rather have a book that's interesting and they feel readers want to read than something with great writing. That's just the truth. Getting published is about more than the writing. It's proof of this when you see so many books (that by some people's standards aren't good writing) but they are huge sellers. Why? They can tell a good story and readers become connected to their voice. Many authors aren't great writers but fantastic storytellers and in today's world, that's what a large group of readers want above all else.
Pubs and agents want books that hook people right off the gate. That first five pages thing is not a myth. It's true because this is also how readers judge a book. In today's quick world, readers don't often have the patience to dive into a book that takes too long to get started. Some do but many live such fast-paced lives now and have so many books at their disposal they check the look inside at Amazon or if still buying print, flip through one or two pages and if they aren't hooked, they don't buy.
That's why agents and pubs are so obsessed with the first few pages. Because this is what determines if a reader will continue or not. If a story is greatly written but agents or editors find it boring, it's not going to sell.
It's great that Hugo people love your work but they might not mirror the interests that everyday readers have for a book. To get an agent or publisher, your book has to have a large enough audience where they figure they can sell it and make some money.
I'm wondering since you said you get such glowing things said about your writing if it is WHAT you are writing that agents and pubs aren't interested in.
Also, if you are getting a lot of form rejections or no response rejections that is a big issue. If many don't even take the time to share thoughts it sounds like your book isn't holding their interest enough to even finish reading unfortunately. There is some disconnect there that you need to try to weed out. Something is stopping them from even replying.
Self-publishing might be a disappointment as well if your audience is too small or you write in a very small niche. I was with a big house, then small houses and now I self-publish so I understand how the methods differ. Self-publishing the right way is HARD and it is even harder if you are writing books that are hard to sell. In that regard, trade publishing
and self-publishing is the same.
You're still going to have to attract the same pool of readers self-publishing that you would with a trade book meaning if the audience is too tiny for a publisher
it will be just as difficult if you self-published. But, if you can get a handle on marketing to your core audience you might can connect.
Another thing, we all can improve no matter what stage we get into. No one is perfect and no book is perfect. It doesn't matter who is telling you your writing is wonderful if agents
and editors aren't interested in it.
Good luck!
I know it's an overused platitude, and it probably won't help much to hear, but just by finishing a book (let alone 5!) you're doing better than most people.
otherwise I see you already have some great advise going here so I'll leave you to it. 
