First, every writer that takes their craft seriously, has a nagging voice of some kind that will try to interrupt the creative/generative process.
The trick is to decide to hell with that self doubting demon and have a ton of fun punching out the story that you think kicks. If you have a blast writing it, others will probably enjoy the ride too. C.S. Lewis had more fun in Narnia than he told in the stories, and that's with going back six more times. Ditto for Rowling, Tolkein and lest you think only fantasy writers get all the fun, Dickens, Austen and O'Connor lost themselves in the worlds they created and loved every minute of it, and thus so do we.
Worry what other people will think when you have the luxury of being able to say, I'm marketing my book or I've just finished my book or even better, I've just published my book and it's going back for a second reprint, would you like me to sign...
In the grudging sludging part of writing my book, love parts, hate parts, need to write the stinking middle...but it's a cool book...I know it.