PDA

View Full Version : Rules for web robots



Sophia
07-27-2010, 11:16 PM
Are the rules for robots visiting AW as all-encompassing as those for the Last.fm site (http://www.last.fm/robots.txt)? :)

Medievalist
07-27-2010, 11:23 PM
We have

Disallow: /trolls
Disallow: /jerks

and

Allow: /bannanation
Allow: /coventry

Yeshanu
07-27-2010, 11:26 PM
We have

Disallow: /trolls
Disallow: /jerks

and

Allow: /bannanation
Allow: /coventry

Um, are you saying we don't have this one:


Disallow: /harming/humans


:gone:

Medievalist
07-27-2010, 11:28 PM
Those are in addition.

We also have a Are You Human? verification test.

quickWit
07-27-2010, 11:32 PM
we should allow snacks.
everyone loves snacks.

Medievalist
07-27-2010, 11:38 PM
No crumbs in the server; makes the hamsters edgy.