Re: robot.txt file
| From: | Cameron Simpson | Date: | Sat, 07 Jul 2001 23:04:40 +0000 |
| Subject: | Re: robot.txt file | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-56688@lists.php.net to get a copy of this message | ||
On Sat, Jul 07, 2001 at 03:36:39PM +0800, Mark Lo <mark@3dsources.com> wrote:
| I don't know how to write a robot.txt file. Can anyone send me a
| link or a sample file.
Well, you could read the documentation:
A Standard For Robot Exclusion
http://www.robotstxt.org/wc/norobots.html
Cheers,
--
Cameron Simpson, DoD#743 cs@zip.com.au http://www.zip.com.au/~cs/
You can't wait for inspiration. You have to go after it with a club.
- Jack London