Jason Kottke has an interesting observation: the whitehouse.gov website has got a new robots.txt file. That's the file that dictates what folders web bots crawling the web to compile search engine lists are not allowed to enter. The old one had 2.400 lines of folders it disallowed. The new one doesn't have any.


Post a Comment

<< Home