The real trick isn't about giving up on a book, it's about writing the next book as soon as possible. Too many new writer put all their energy into trying to find an agent and publisher for a book. Yes, keep the book out there. If it's any good at all, someone, somewhere will take an interest in it, and if not in it, then in your potential.
But while that book is making the rounds you must be writing a second, a third, and a fourth, if you're prolific enough. There's no more reason to expect a first novel to sell than to expect a first painting to sell. But you keep it on the market until there simply is no one else to send it to. No more agents, no more editors, no more anything.
But submitting anything is always a second priority. Priority number one must be getting the second, third, however many books you can write finished and on the market. Even a slow writer should be able to have a second novel making the rounds long before all the possible agents and publisher for the first book are exhausted.
You learn with each book. THE novel you need to break through might be the first, but it might also be the fourth or the fifth of the ninth.
Being a writer isn't about any one book, and one project, it's about the process of writing continually, of learning and growing with each new book. You simply can't worry about whatever book or books you have in submission. You just keep sending them out as they get rejected, but you have to concentrate your time, energy, and emotional state on the book you;re writing at the moment, and there should always be a book you're writing at the moment.