Re: Website crawler auto image downloader
| From: | Shawn McKenzie | Date: | Sat, 17 Jan 2009 00:33:47 +0000 |
| Subject: | Re: Website crawler auto image downloader | ||
| References: | 1 2 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-286536@lists.php.net to get a copy of this message | ||
Sam Smith wrote:
> I need a script that will crawl a list of websites and download all .jpg,
> .gif, .png files.
>
> I can think of some ways how to start like, fopen() or maybe curl(). And it
> downed on me I'd need to handle files writing over with the same name. And
> it would be cool to save the full URL to the file in a database with the
> path on the local server where I'm saving them.
>
> I was hoping someone might say, "Dude, that's simple, just do this..."
> before I spent hours guessing.
>
> Anyone?
>
> Thanks
>
>
Dude, that's simple, just do this...
- crawl the page extracting the src of images, regex or dom
- fopen and read it
- fput and save it, if it exists add an incremental number
- save url and local dir/filename in db
--
Thanks!
-Shawn
http://www.spidean.com