Re: PHP web spyder?
| From: | R. P. Channing Rodgers, M.D. | Date: | Thu, 27 Jul 2000 21:26:47 +0000 |
| Subject: | Re: PHP web spyder? | ||
| Groups: | php.general | ||
| Request: | Send a blank email to php-general+get-8671@lists.php.net to get a copy of this message | ||
On UNIX systems, the GNU wget utility is often used for this:
ftp://ai.prep.mit.edu/pub/gnu/wget*
> From: "Marc Swanson" <marcswanson@mediaone.net>
> Date: Thu, 27 Jul 2000 17:14:51 -0400
> Subject: [PHP] PHP web spyder?
>
> Before I spend a lot of my time coding my own web spyder, I'd thought I'd
> ask if anyone has already done this as an open source project?
>
> What I need is something that will follow all links on a supplied URL up to
> a specified debth and store them either on the filesystem or in a db
> including images on the pages.
>
>
> Thanks
>
> -Marc-