New Message: Spammers and slow page loads

[webmaster at userland.com]

A new message was posted:

Address: http://frontier.userland.com/discuss/msgReader$11664

By: daniel springett (ds at dp.co.nz)

Bit of a heads up for those with servers noticing slow page loads and high thread counts.

Clients have been seeing very slow page load times, and myself with thread counts up in the low 30s on my server.

Checked the server log files and see that some sites (porn) are targeting /robots.txt and then the stats/referers pages, hitting the server so often that the server maxes out at 32 threads and bogs down.

It is I think an attempt to use the "referer" logs that many websites now display to draw in viewers and influence the search engines.

My own referers page at http://www.dp.co.nz/stats/referers is a good example. the search engine referers are showing these recent requests

My server (sites) are near top of the results for some of these search terms

As the offending referers are many domains its is very hard if not impossible to block at the isp level

64.255.xxx.xxx

64.255.30.16

64.255.30.17

64.255.28.16

But not all 64.255 sites are offenders

I do not know what to do other than attempt to block at the isp incoming level, but as the offenders are using multiple domains it is very difficult without a global block.

I would guess that any attempt at the Mainrsponder level would also bog the server down as the request would need to be "handled".

Am removing all references to "stats/referers" from the sites I "control".

How do I "turn off" the stats/referers on individual sites and globally

Any other suggestions would be welcomed.

http://manila.userland.com/stats/referers

Appears to have a similar issue with search terms

Cheers

Daniel Springett

This is a Manila site.. http://manila.userland.com/.