slow down search engine spiders

If you have a large or busy website you may want to slow down the pace that search engine bots scan your site, there's a growing number of bots and when several get onto your site at once they can impact performance and plus they're using your bandwitdh.

To slow down search engine bots you need to add the following to a robots.txt file in your public_html:
User-agent: *
Crawl-delay: 60

You can slow down the speed that google search engine bots hit your site by using the Google Webmaster tools:
http://www.google.com/webmasters/

  • 17 Users Found This Useful
Was this answer helpful?

Related Articles

I think my site is down, what should I do?

If you think that your website or server is down please give at least 10 minutes before...

Do you provide SSH/Telnet access with your Web Hosting/Reseller Hosting accounts?

Sorry but SSH/Telnet access is not provided with our Web Hosting / Reseller Hosting accounts. If...

How can I disable directory browsing?

You can disable directory browsing from your cPanel control panel using the "Index Manager", you...

What is the path to my web directory and other utilities?

The path to your web directory is: /home/username/public_htmlReplace "username" with the actual...

How do I setup secure webspace and the SSL certificate?

To use SSL encryption on your web hosting account you will need two things:1. SSL CertificateFor...