Suggestions, Comments and Site Questions Post your comments and suggestions here. We always want to improve the site.

Googlebot, our little tag along, friend?

Thread Tools
 
Search this Thread
 
Old Jan 26, 2005 | 08:20 AM
  #1  
BoostdCTD's Avatar
Thread Starter
Registered User
 
Joined: Aug 2004
Posts: 847
Likes: 0
From: Houston, TX
Googlebot, our little tag along, friend?

Hi all,
I know the DTR server is getting pushed to the max and today when I was at my forum I noticed that something kept following me around. So I took the IP down and looked it up, it traced back as a "googlebot." No I don't have any viruses or anything like that, apparently the way web searches get info is by sending a web crawler around with people. So from what I read the web crawler goes onto every page that someone goes then starts clicking on every single link on that page and archives all of it. Well if you have 900 people that would be a lot of extra activity. Here is some more information here. http://www.google.com/bot.html I'm not sure how applicable this problem is to putting a load on the server but wouldn't that be cool if rather than disabling the search feature under traffic we could disable the webcrawler after so many people are here, just a thought....

Why is Googlebot downloading the same page on my site multiple times?

In general, Googlebot should only download one copy of each file from your site during a given crawl. Occasionally the crawler is stopped and restarted, and it may recrawl pages that it has recently retrieved. These recrawls should happen infrequently.
Reply
Old Jan 26, 2005 | 08:26 AM
  #2  
BoostdCTD's Avatar
Thread Starter
Registered User
 
Joined: Aug 2004
Posts: 847
Likes: 0
From: Houston, TX
If you choose to try something along these lines or just research how much of a load the web crawlers cause, this may be the solution.
Reply
Related Topics
Thread
Thread Starter
Forum
Replies
Last Post
D.S.
Towing and Hauling / RV
17
Dec 22, 2008 11:02 PM
2500Ram
Other
18
Sep 13, 2005 08:16 AM
Haulin_in_Dixie
Towing and Hauling / RV
39
Jul 17, 2005 08:25 PM
csutton7
Other
64
Dec 24, 2003 01:39 AM




All times are GMT -5. The time now is 06:25 AM.