Googlebot, our little tag along, friend?
01-26-2005, 08:20 AM
#1
Registered User
Thread Starter
Join Date: Aug 2004
Location: Houston, TX
Posts: 847
Likes: 0
Received 0 Likes
on
0 Posts
Googlebot, our little tag along, friend?
Hi all,
I know the DTR server is getting pushed to the max and today when I was at my forum I noticed that something kept following me around. So I took the IP down and looked it up, it traced back as a "googlebot." No I don't have any viruses or anything like that, apparently the way web searches get info is by sending a web crawler around with people. So from what I read the web crawler goes onto every page that someone goes then starts clicking on every single link on that page and archives all of it. Well if you have 900 people that would be a lot of extra activity. Here is some more information here. http://www.google.com/bot.html I'm not sure how applicable this problem is to putting a load on the server but wouldn't that be cool if rather than disabling the search feature under traffic we could disable the webcrawler after so many people are here, just a thought....
I know the DTR server is getting pushed to the max and today when I was at my forum I noticed that something kept following me around. So I took the IP down and looked it up, it traced back as a "googlebot." No I don't have any viruses or anything like that, apparently the way web searches get info is by sending a web crawler around with people. So from what I read the web crawler goes onto every page that someone goes then starts clicking on every single link on that page and archives all of it. Well if you have 900 people that would be a lot of extra activity. Here is some more information here. http://www.google.com/bot.html I'm not sure how applicable this problem is to putting a load on the server but wouldn't that be cool if rather than disabling the search feature under traffic we could disable the webcrawler after so many people are here, just a thought....
Why is Googlebot downloading the same page on my site multiple times?
In general, Googlebot should only download one copy of each file from your site during a given crawl. Occasionally the crawler is stopped and restarted, and it may recrawl pages that it has recently retrieved. These recrawls should happen infrequently.
In general, Googlebot should only download one copy of each file from your site during a given crawl. Occasionally the crawler is stopped and restarted, and it may recrawl pages that it has recently retrieved. These recrawls should happen infrequently.
Thread
Thread Starter
Forum
Replies
Last Post