cvs: php-news-web / robots.txt
| From: | Rasmus Lerdorf | Date: | Wed, 01 Mar 2006 01:48:37 +0000 |
| Subject: | cvs: php-news-web / robots.txt | ||
| Groups: | php.mirrors | ||
| Request: | Send a blank email to php-mirrors+get-30751@lists.php.net to get a copy of this message | ||
rasmus Wed Mar 1 01:48:37 2006 UTC
Modified files:
/php-news-web robots.txt
Log:
Let's try letting Slurp in as long as it crawls very slowly so our new
search mechanism can get at this as well.
http://cvs.php.net/viewcvs.cgi/php-news-web/robots.txt?r1=1.1&r2=1.2&diff_format=u
Index: php-news-web/robots.txt
diff -u php-news-web/robots.txt:1.1 php-news-web/robots.txt:1.2
--- php-news-web/robots.txt:1.1 Fri Nov 26 12:16:26 2004
+++ php-news-web/robots.txt Wed Mar 1 01:48:37 2006
@@ -1,2 +1,6 @@
User-agent: *
Disallow: /
+
+User-agent: Slurp
+Crawl-delay: 5
+Disallow: