Yes you can reboil it, but what I would do is stick it in the fridge for a few days to see if it does set - sometimes it takes longer. If you do reboil it be prepared to have less, since more liquid will evaporate concentrating the product.
What kind of jam are you making? The reason I ask is that some fruit have more natural pectin in them, making a commercial pectin product unnecessary. Crabapple and apple and sour fruits like gooseberry or currents have a high amount of natural pectin so you don't need any, but softer fruits like strawberry, peach etc you do.
I have a candy thermometer which I use to make sure that the mix has reached the proper temperature for jelly and then I start testing (some fruits take longer than others). Start by sticking your spoon in the boiling liquid, removing it hold it vertical, watch the drops fall. One indication is if the jelly is falling in a steady stream it's not ready, but once the last few drops on the spoon start falling slower and two drops bead together then it most likely is done.
Another test is to take it from the heat and let it sit for a moment, run your spoon lightly over the surface and if it picks up a "skin" then it's more than likely done.
You could also stick a saucer in the freezer and when you think that it may be done, pour a teaspoon (from a bit of a height) on it, let it sit for a moment on the cold plate and then tilt the plate - if the edge of the pool has a "rim" then it's good to go. I often use all three methods.
Don't give up, and even if it never gels properly, it will make a beautiful syrup for ice cream or crepes/pancakes.