Re: building a "crawler" with PHP
| From: | Miguel Cruz | Date: | Tue, 14 May 2002 02:16:29 +0000 |
| Subject: | Re: building a "crawler" with PHP | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-97464@lists.php.net to get a copy of this message | ||
On Mon, 13 May 2002, Robert Mena wrote:
> Hi, I'd like to know if anyone has good references
> (links, ps, pdf etc) about building a crawler and tips
> regarding doing that with PHP.
>
> This would be mostly for learning purposes.
I'll just give you one tip:
Keep track of your stack size! It's easy to follow looping links into
accursed recursive oblivion.
Other than that, not much to it. Global array of links. Recursive function
to fetch all the links on a page, add them to the global array (if not
already present), and then call itself on each one.
miguel