Googlebot, our little tag along, friend?
Googlebot, our little tag along, friend?
Hi all,
I know the DTR server is getting pushed to the max and today when I was at my forum I noticed that something kept following me around. So I took the IP down and looked it up, it traced back as a "googlebot." No I don't have any viruses or anything like that, apparently the way web searches get info is by sending a web crawler around with people. So from what I read the web crawler goes onto every page that someone goes then starts clicking on every single link on that page and archives all of it. Well if you have 900 people that would be a lot of extra activity. Here is some more information here. http://www.google.com/bot.html I'm not sure how applicable this problem is to putting a load on the server but wouldn't that be cool if rather than disabling the search feature under traffic we could disable the webcrawler after so many people are here, just a thought....
I know the DTR server is getting pushed to the max and today when I was at my forum I noticed that something kept following me around. So I took the IP down and looked it up, it traced back as a "googlebot." No I don't have any viruses or anything like that, apparently the way web searches get info is by sending a web crawler around with people. So from what I read the web crawler goes onto every page that someone goes then starts clicking on every single link on that page and archives all of it. Well if you have 900 people that would be a lot of extra activity. Here is some more information here. http://www.google.com/bot.html I'm not sure how applicable this problem is to putting a load on the server but wouldn't that be cool if rather than disabling the search feature under traffic we could disable the webcrawler after so many people are here, just a thought....
Why is Googlebot downloading the same page on my site multiple times?
In general, Googlebot should only download one copy of each file from your site during a given crawl. Occasionally the crawler is stopped and restarted, and it may recrawl pages that it has recently retrieved. These recrawls should happen infrequently.
In general, Googlebot should only download one copy of each file from your site during a given crawl. Occasionally the crawler is stopped and restarted, and it may recrawl pages that it has recently retrieved. These recrawls should happen infrequently.
Thread
Thread Starter
Forum
Replies
Last Post
Haulin_in_Dixie
Towing and Hauling / RV
39
Jul 17, 2005 08:25 PM



