Re: a loop constructing the URLs and make PHP to fetch up to 10 thousand pages

On 23 October 2010 06:10, "jobst müller" <Floobee@web.de> wrote: > hello dear list - good morning! > > > I am trying to figure out a method how to retrieve on the same URLs [see  below] with > different query arguments, and i am wondering if this is doable with PHP!? > > on a sidenote:  Well - i guess that we can do this with  LWP::UserAgent -  guess that this > provides a way for us to loop through the query arguments: I am not sure that LWP::UserAgent has a > method for us to do that. I tried to figure it out. And i digged deeper in the Manpages and Howtos. > we can have a loop constructing the URLs and use > LWP::UserAgent repeatedly: > > > see the Code: > > for my $id (0 .. 100000) > > { > >  $ua->get($url."?id=21&extern_eid=".(0-$id)) > >  //rest of the code > > } > > > > Well, alternatively we can add a request_prepare handler that  computes and add the query > arguments before we send out the request. Do you think that this fits the needs? > > But wait - i want to do this with PHP! > > What is aimed: Here on this following site we find a list of many  schools: [see the page with > the subsequent results - approx more than 1000 sites] > > see this site:  http://www-db.sn.schule.de/index.php?id=25 > > i want to fetch the sites that are listet on this page - and therefore i  want to use PHP for > this [job] - and subesquently >  i want to parse them. > > the sites can be reached directly - by constructing in other words the subsites of the overview > can be reached via direct >  links... see the following. > > > http://www-db.sn.schule.de/index.php?]id=21&extern_eid=1543 > > http://www-db.sn.schule.de/index.php?]id=21&extern_eid=709 > > http://www-db.sn.schule.de/index.php?]id=21&extern_eid=789 > > http://www-db.sn.schule.de/index.php?]id=21&extern_eid=1297 > > > > Well - i want to fetch all those.... And i try to do it with PHP and a > mentioned loop. Does this work!? > ___________________________________________________________ > GRATIS! Movie-FLAT mit über 300 Videos. > Jetzt freischalten unter http://movieflat.web.de > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > You can go down that route. If the site has an API allowing you to gather the data another way (i.e. in 1 large file), then use that instead. I'd also make sure you are allowed to gather all that data. You may find some issues with that - just check to be sure. Also, you had a typo in the URLs ... an extra ] for no reason. -- Richard Quadling Twitter : EE : Zend @RQuadling : e-e.com/M_248814.html : bit.ly/9O8vFY