Whatever you do, I would suggest you always ensure you are acting in good faith. Even if you don't like them or their methods, you don't want to murky the waters by being anything but 100% professional.
Just send the advance back to them, thank them for their time and terminate the relationship. Perhaps indicate that it isn't yet at a professional level so they would be wasting money to publish it at this time.
Just as a minor note, there is a track record of publishers getting the advance back when the author decides not to continue with publication - and it seems only reasonable. (I'm thinking of a case earlier this year of a tell-all expose book that theauthor backed out on, leaving the publisher high and dry. In that case, the publisher had done a heck of a lot of work on legal fees, ARCs editing etc ... and all they did was get the advance back. I felt sorry for the publisher.)
Mac
