RE: [PHP] robot.txt file

if you want to know when you are spidered write yourself a little function that logs some data from the visitor. For instance you will need at least HTTP_USER_AGENT, REQUEST_URI and current time. Write that all in your db and then periodically go and look into it. You will see if the spiders are trying to read robots.txt and what they do once they read it. I almost agree with Don, not all spiders are following the instructions you gave them, but some do. It makes sense to spiders following the instructions, as if the page is not the one you want people to see their own database will just get polluted by some crap as well as the bandwidth is not any cheap to go crawling the crap. IBM's spider is, however, the worth I've ever seen. It literally "guesses" your urls. PHPBeginner.com is driven by apache's mod_rewrite, so any page you go you will find the frontpage, unless the URL makes some sense for my application and can be translated into variables to pull the right article out. When IBM's spider comes, it crawls me pages like 'about.html', 'contact.html', 'default.html' and even '_index.html' and 'index2.html'... I never had such files on my site, but they result him fine - the front page. IBM guessed, crawled and 'got happy'! Just to explain you what these damn crawlers like to do :-) Sincerely, Maxim Maletsky Founder, Chief Developer PHPBeginner.com (Where PHP Begins) maxim@phpbeginner.com www.phpbeginner.com -----Original Message----- From: Tom Carter [mailto:subs@roundcorners.com] Sent: Saturday, July 07, 2001 4:50 PM To: Don Read Cc: php-general@lists.php.net Subject: Re: [PHP] robot.txt file Hi Don, How do you work this out? ie how do you know when you are spidered? I'm interested in what logs and what patterns lead you to know this? Thanks, Tom > > On 07-Jul-01 Mark Lo wrote: > > Hi, > > > > I don't know how to write a robot.txt file. Can anyone send me a > > link or a sample file. > > > > that's 'robots.txt' > > User-Agent: * > Disallow: /images/ > Disallow: /icons/ > Disallow: /cgi-bin/ > Disallow: /php/ > Disallow: /tmp/ > Disallow: /corp/login.html > Disallow: /cust/login.html > > --- > But don't expect it to do anything for you, it's just a 'suggestion' to > well-behaved spiders. > > My observation: > 50% of them check & obey the file, 45% don't even check, & some 5% check > then try the 'prohibited' URLs > > Regards, > -- > Don Read dread@texas.net > -- It's always darkest before the dawn. So if you are going to > steal the neighbor's newspaper, that's the time to do it. > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net > For additional commands, e-mail: php-general-help@lists.php.net > To contact the list administrators, e-mail: php-list-admin@lists.php.net > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net For additional commands, e-mail: php-general-help@lists.php.net To contact the list administrators, e-mail: php-list-admin@lists.php.net