Re: robot.txt file

Hi Don, How do you work this out? ie how do you know when you are spidered? I'm interested in what logs and what patterns lead you to know this? Thanks, Tom > > On 07-Jul-01 Mark Lo wrote: > > Hi, > > > > I don't know how to write a robot.txt file. Can anyone send me a > > link or a sample file. > > > > that's 'robots.txt' > > User-Agent: * > Disallow: /images/ > Disallow: /icons/ > Disallow: /cgi-bin/ > Disallow: /php/ > Disallow: /tmp/ > Disallow: /corp/login.html > Disallow: /cust/login.html > > --- > But don't expect it to do anything for you, it's just a 'suggestion' to > well-behaved spiders. > > My observation: > 50% of them check & obey the file, 45% don't even check, & some 5% check > then try the 'prohibited' URLs > > Regards, > -- > Don Read dread@texas.net > -- It's always darkest before the dawn. So if you are going to > steal the neighbor's newspaper, that's the time to do it. > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net > For additional commands, e-mail: php-general-help@lists.php.net > To contact the list administrators, e-mail: php-list-admin@lists.php.net >