It's not a rule. Adverbs are useful in many cases, to get a point across. The problem with adverbs is that some writers tend of overuse them:
"I'm leaving now," she said angrily. Then she walked away briskly. She quickly stopped at the bakery and fondly remembered going there as a little girl.
Another problem is that it's lazy writing -- many of the verb-adverb pairs can be replaced with stronger verbs, so why does the writer not write more strongly [see, I used an adverb here]? Also, adverbs are by nature "tell" instead of "show." We're saying she said "angrily" without showing us how she is angry.
Let's look at the sentence above:
1. "angrily" can be shown more effectively.
2. "walked away briskly" can be replaced by a stronger verb
3. "quickly stopped" is redundant -- simply "stopped" would be fine
4. "fondly remembered" is telling and cliched, plus it's probably better rewritten as "fond memories [verb] her..."
Again, adverbs are not all bad. I used a couple in this post. They add flavors. Obviously every adverb can be replaced by something else, but if you do that, you may stiffen your prose. Still, you need to scrutinize every adverb you use and see if it should exist.