MSN bot sucking bandwidth

16 Feb 2005 12:14 - (2) comments

MicroSoft's new search bot really is becoming a nuisance. I'm having slower and slower server respones. The bot really likes this demo of a bookkeeping application. In the application you can browse through all the years you want. But because there is only some small sample data for the current months, you won't find anything interesting on other dates.

MSN bot isn't so sure. So I added the following to the robots.txt:


Disallow: /demo/

User-agent: msnbot

Crawl-delay: 900

With the Crawl-delay MSN bot should only visit my site every 15 minutes.
Let's see if this helps.

Comments

And did it help?

On 17 Feb 11:11 by Rod

Well, it seems the slow responses were due to me uploading a large file. :)

On 17 Feb 11:47 by Petrik

Comments have been closed.

Admin