I think there is a line, and it does have something to do with the crowd, but that's not the only thing.
Some humour gets laughs by grossing people out, and that offends some people. But it's still humourous. But as Perks says, you should avoid setting yourself up to be offended. If you don't like gross or sexual humour, don't hang out with people who use that kind of humour.
Where I feel humour can cross the line is where it is really hate disguised as humour. That's why I have trouble with jokes aimed at specific targets, like blondes or people of colour. I ask myself, would this joke be funning if it was about a different group/type of people. If the answer is no, then the joke is probably hate rhetoric disguised as humour.
If the answer is yes, then it's probably a funny joke. For example, light bulb jokes change targets depending on the audience/comedian. They illustrate the comic side of belonging to a particular group. For example:
How many second violinists does it take to screw in a light bulb?
None; second violinists can't reach that high.
How many saxophone players does it take to screw in a light bulb?
Five; one to screw in the light bulb and four to stand around and wonder how David Sanborn would do it.
How many percussionist does it take to screw in a light bulb?
Two; one to screw in the light bulb after the other realizes that hitting it with a stick won't work.
How many 1st Bassoonists does it take to change a light bulb?
None...that's the 2nd bassoonist job!
How many oboists does it take to change a light bulb?
One, but it takes them an hour to find just the RIGHT bulb...
How many clarinet players does it take to screw in a light bulb?
One, but it will take him a huge box of light bulbs to find one that fits.
How many producers does it take to change a light bulb?
Well, what do you think?
How many sound men does it take to change a light bulb?
One, two, three, one, two, three.
Obviously, one should avoid setting oneself up to be offended. I'm very difficult to offend, so if I find myself there, I usually ask myself two quick questions: "Was there intent?" and "Do I really give a sh**?" Those two can generally un-bunch my panties in a hurry...
This is about right. Intent to offend or to inspire hate makes the joke not funny. If I answer the second question with a "Yes, I do give a sh**," then I'll simply tell the person, "That wasn't funny," and walk away. Comedians live for laughs. If they don't get them, I find they mend their ways pretty fast...