Re: htdig
| From: | Adolfo Garcia Veytia | Date: | Sat, 20 Apr 2002 08:15:32 +0000 |
| Subject: | Re: htdig | ||
| References: | 1 2 | Groups: | php.mirrors |
| Request: | Send a blank email to php-mirrors+get-9691@lists.php.net to get a copy of this message | ||
Well, perhaps you are indexing external sites, check out
the limit_urls_to parameter on the configuration file.
Maybe it is not properly set and the robot is indexing
web pages from other servers.
At 08:51 4/20/2002 +0200, Trond Arve Nordheim wrote:
On Sat, Apr 20, 2002 at 09:27:25AM +0300, Spyros Ioakim wrote: 1) How big is the htdig database after completion? (I have 44GB on another filesystem so maybe move it there) Depends entirely of the size of my php-mirror htdig database is 1.7GB. The web itself is 869MB, including tarballs and zip files (wow, never noticed that difference before now ;) 2) How long does it take for indexing to be completed and is htdig that smart so that it won't go through the whole site every night? It will probably check all files, but only index those who changed. -- Trond Arve Nordheim - "This message is ROT13-encrypted twice for extra security."Adolfo Garcia Investigacion y Desarrollo - Urbano PGP Fingerprint 804C 5DCB 4772 4A87 36EC 117A 761E 34C9 640C 8BCF