RE: [PHP] robot.txt file
| From: | Don Read | Date: | Sat, 07 Jul 2001 07:46:33 +0000 |
| Subject: | RE: [PHP] robot.txt file | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-56618@lists.php.net to get a copy of this message | ||
On 07-Jul-01 Mark Lo wrote:
> Hi,
>
> I don't know how to write a robot.txt file. Can anyone send me a
> link or a sample file.
>
that's 'robots.txt'
User-Agent: *
Disallow: /images/
Disallow: /icons/
Disallow: /cgi-bin/
Disallow: /php/
Disallow: /tmp/
Disallow: /corp/login.html
Disallow: /cust/login.html
---
But don't expect it to do anything for you, it's just a 'suggestion' to
well-behaved spiders.
My observation:
50% of them check & obey the file, 45% don't even check, & some 5% check
then try the 'prohibited' URLs
Regards,
--
Don Read dread@texas.net
-- It's always darkest before the dawn. So if you are going to
steal the neighbor's newspaper, that's the time to do it.