You say you have changed the names, location etc (and how about changing the dates a couple of years earlier/later?) Why not publish it under a pseudonym, adding another layer of defences against anyone ferreting out the true identities? Use the 'true' version merely as your springboard, making fiction out of the material you already have.
You say you cannot reach the character -perhaps the trouble-makers can't find them either.
Plus, as we all know, not everyone buys or reads our books! You can always discuss this further when you meet your agent in the future.