[web-news] master: Allow search engines to crawl the archives
| From: | Tim Düsterhus via Derick Rethans | Date: | Thu, 24 Sep 2026 08:05:24 +0000 |
| Subject: | [web-news] master: Allow search engines to crawl the archives | ||
| Groups: | php.webmaster | ||
| Request: | Send a blank email to php-webmaster+get-34045@lists.php.net to get a copy of this message | ||
Author: Tim Düsterhus (TimWolla)
Committer: Derick Rethans (derickr)
Date: 2026-09-24T09:05:21+01:00
Commit: https://github.com/php/web-news/commit/597a2b84e2394ffab48e3a522596c9f964798961
Raw diff: https://github.com/php/web-news/commit/597a2b84e2394ffab48e3a522596c9f964798961.diff
Allow search engines to crawl the archives
see php/infrastructure#40
Changed paths:
M robots.txt
Diff:
diff --git a/robots.txt b/robots.txt
index 8936d2c..d08e401 100644
--- a/robots.txt
+++ b/robots.txt
@@ -1,6 +1,2 @@
User-agent: *
-Disallow: /
-
-User-agent: bingbot
Crawl-delay: 5
-Disallow: