Re: PHP web spyder?
| From: | Jesper Juhl | Date: | Fri, 28 Jul 2000 17:40:40 +0000 |
| Subject: | Re: PHP web spyder? | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-8824@lists.php.net to get a copy of this message | ||
> Before I spend a lot of my time coding my own web spyder, I'd thought I'd
> ask if anyone has already done this as an open source project?
> What I need is something that will follow all links on a supplied URL up
to
> a specified debth and store them either on the filesystem or in a db
> including images on the pages.
> Thanks
> -Marc-
Wouldn't wget (http://www.gnu.org/software/wget/wget.html) be useable for
that purpose (and you probably already have it if you are using Linux or
FreeBSD)?
/Jesper Juhl
juhl@eisenstein.dk