New Message: Re: Prefs>Advanced>Robots.txt

[webmaster at userland.com]

A new message was posted:

Address: http://manila.userland.com/discuss/msgReader$2985

By: Steve Hooker (steve at cybersaps.com)

I've edit the script at manilasuite.robots.robotstxt. In this way all sites are affected. Warning, editing such may be over wiped by an update from UserLand.

I run lots of custom scripts here. There are several in my config.mainResponder.Callbacks.controlAccess. Sorry, not standard. Thought it would be useful to note that though Googlebot is probably the biggest, most active crawler, there are to my knowledge nearly 400 possibly worse offenders. I've see them come in at such a speed, with so many requests per second that my server comes to its knees. When spotted they're banned. But also terrible bandwidth devils are the single users who, "save this site, locally." OH! MY Head!

Steve Hooker

http://www.blogfrootball.com/service

http://www.walsallschools.org/

This is a Manila site... http://manila.userland.com/.