It's not uncommon to have an Idea File, a folder on your computer (or an actual, physical file or drawer if you write longhand) for the unfinished, the fragmentary, the stray idea jotted down on a cocktail napkin that you haven't gotten around to exploring. If you're working on Story A and you get an inspiration about Story B or an idea that could be Story C, you can stick them in the Idea File so you don't get derailed from Story A; they'll wait for you.
To avoid having that file explode to eat all available space (disk or otherwise), try to check in on it every so often. Do not be afraid to cull those that no longer speak to you. As they say, ideas are a dime a dozen. It's the execution that makes it a story, and not all ideas can or ever will be executed by any given writer.
Now, if it's a habit of yours, collecting story starts and abandoning them when the going gets a little rough and a new shiny idea races across the path like a rabbit in a waistcoat, that comes down to issues of self-discipline and focus and learning how (and when) to stick it out and work through problems.