Thanks, CaroGirl and Silver King. The "of" is not really important here. The sentence could just as easily be: I have a doctors appointment on Thursday.
This is how people commonly speak where I am from in California, so I don't want to change the phrase. But I just can't seem to decide if it should be doctor/doctors/doctor's appointment, although I do agree with Silver King that it should not be possessive. At the same time, I am almost sure I hear people say it with an s (doctors), yet "doctors appointments" looks strange to me.