I would, especially with gestures. I doubt there is a specific optimum number, but it would be related to the number of uses compared to the number of scenes that the "tagged" character is in, length of the novel, number of characters for the reader to keep track of, etc.

In the example I gave about the character tugging his ear (from Real Life & Liars, an otherwise good novel), it was in a secondary character, the only son of the MC who had 3 or 4 grown kids who all had about equal weight in the story. The fact that he was the only son made him memorable enough within the cast, and he was well very drawn and consistent as a character. But he tugged his ear in every single scene he was in. Even if he was in only 4 scenes, that would have been too many ear tugs for him, IMO.

So I'm going to squint at my thumb in the air and say keep it between 20% and 30% of the scenes the character appears in. I just made that up; please don't take it as any kind of rule.