a loop constructing the URLs and make PHP to fetch up to 10 thousand pages
| From: | "jobst müller" | Date: | Sat, 23 Oct 2010 05:10:05 +0000 |
| Subject: | a loop constructing the URLs and make PHP to fetch up to 10 thousand pages | ||
| References: | 1 2 3 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-309034@lists.php.net to get a copy of this message | ||
hello dear list - good morning!
I am trying to figure out a method how to retrieve on the same URLs [see below] with different
query arguments, and i am wondering if this is doable with PHP!?
on a sidenote: Well - i guess that we can do this with LWP::UserAgent - guess that this provides
a way for us to loop through the query arguments: I am not sure that LWP::UserAgent has a method for
us to do that. I tried to figure it out. And i digged deeper in the Manpages and Howtos. we can have
a loop constructing the URLs and use
LWP::UserAgent repeatedly:
see the Code:
for my $id (0 .. 100000)
{
$ua->get($url."?id=21&extern_eid=".(0-$id))
//rest of the code
}
Well, alternatively we can add a request_prepare handler that computes and add the query arguments
before we send out the request. Do you think that this fits the needs?
But wait - i want to do this with PHP!
What is aimed: Here on this following site we find a list of many schools: [see the page with the
subsequent results - approx more than 1000 sites]
see this site: http://www-db.sn.schule.de/index.php?id=25
i want to fetch the sites that are listet on this page - and therefore i want to use PHP for this
[job] - and subesquently
i want to parse them.
the sites can be reached directly - by constructing in other words the subsites of the overview can
be reached via direct
links... see the following.
http://www-db.sn.schule.de/index.php?]id=21&extern_eid=1543
http://www-db.sn.schule.de/index.php?]id=21&extern_eid=709
http://www-db.sn.schule.de/index.php?]id=21&extern_eid=789
http://www-db.sn.schule.de/index.php?]id=21&extern_eid=1297
Well - i want to fetch all those.... And i try to do it with PHP and a
mentioned loop. Does this work!?
___________________________________________________________
GRATIS! Movie-FLAT mit über 300 Videos.
Jetzt freischalten unter http://movieflat.web.de