Re: robot.txt file
| From: | Michael H. Warfield | Date: | Sat, 07 Jul 2001 08:32:27 +0000 |
| Subject: | Re: robot.txt file | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-56623@lists.php.net to get a copy of this message | ||
On Sat, Jul 07, 2001 at 03:36:39PM +0800, Mark Lo wrote:
> Hi,
> I don't know how to write a robot.txt file. Can anyone send me a
> link or a sample file.
Real simple... Just enumerate all the places you want agents
to NOT go...
User-agent: *
Disallow: /icons/
Disallow: /backgrounds/
Disallow: /images/
Disallow: /cci/
Disallow: /webservice/
Disallow: /stats/
Disallow: /admin/
Disallow: /linux/
Disallow: /docs/
Disallow: /tmp/
Disallow: /private/
Disallow: /smythe/
Disallow: /magic_n_mirrors/
You can get more complicated than that, but something like that
will do most of what you want to do.
> Thank you for your help
> Mark Lo
Mike
--
Michael H. Warfield | (770) 985-6132 | mhw@WittsEnd.com
(The Mad Wizard) | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!