Hopefully the stereotype does come from folks who were isolated - backwoods can mean quite isolated. Then it makes more sense that inbreeding would occur.
In my paternal family line, starting with a pre-revolution settlement to just before the 1900's, the family married first cousins more often than not. It was a German religious settlement, and they really didn't approve of outsiders. Originally there weren't many people around [actually, there were a ton of Indians], but later that excuse didn't hold water.
I was born with a tail

That would be an awesome freaky-inbreeding one, although B vitamins were more likely to blame, really. It's called Spina Bifida Occulta and can include great freakiness in its various forms (it can be quite sad, too, of course, but I'm just talking fiction possibilities).
It has a genetic component, at least once introduced in a line.