Re: Manuals and Spiders

From: Date: Wed, 23 Apr 2003 18:00:43 +0000
Subject: Re: Manuals and Spiders
References: 1  Groups: php.mirrors 
Request: Send a blank email to php-mirrors+get-16958@lists.php.net to get a copy of this message
I think a cron is a good idea - at least to start off this project. I can do the fixing at our end, you just need to define the exact format that you want. What we will do, is give you a url like http://www.directi.com/iptocountry.php?site=php.net&dbformat=1&lastupdat e=unixtimestamp to update the db. This will give you the entire db in the exact format that you want. This way the fixing code does not need to be there on every mirror. The db is fixed as per php.net's requirements on our server itself. Also, if we ever update the db in a different format - in case we start giving more details than what exists in the current db - we would just need to update the script at a single location. This will ensure that php and all its related sites would always get the db in the correct format.
Well, we will get your database on rsync.php.net and redistribute it to the mirrors using rsync (you know, we does not require any of our mirrors to have url_fopen feature turned on / built into PHP). So we would be able to do the needed conversion [kind of scraping], but anytime you would modify the data format, we would be in trouble ;) So it is better to define the format on your end, and get the proper data...
Just give me a sample of the format that you want - by changing 2-3 lines of the current db as per the new format and sending it across to me / on the list. Once I have this, we can easily write a script to convert the current db format to the one you want. We will then provide you with the url that you can put in the cron to get the db automatically. The params to be included in the update url are as below. Let me know if any one can think of any more essential params, or if anyone thinks a particular param should not exist: http://www.directi.com/iptocountry.php?site=php.net&dbformat=1&lastupdat e=unixtimestamp Site - would tell us which group of sites is requesting this data. So all php related websites would send this as php.net Dbformat - in case we start giving additional params that you want to use, we just need to update the version number. This will allow us to maintain a single script for all formats. We will provide the update as per the format requested. Lastupdate - we can return a "no update", in case there is no updated db since the last update date. The cron on the server should call a local php script which will make a call to this URL. In case php decides to change the dbformat then this local script is updated using rsync. The cron does not need to change for ever.
OK, first of all we need to find out if anyone has objectsions against adding this. Especially James or Jim may have comments on this. We need to wait a bit for them to provide their opinion. Then first before giving you work to output the database in the format defined, it would be good to write a simple test script to search in a sample file using that format. If it turns out that the format is not all right, then you worked for nothing... Only after this can we go on to add that cronjob IMHO. James/Jim, is there a base of backend data existing on rsync.php.net, so we can use the old ip2country data if available, or is the rsync servers data cleaned before the update scripts are executed and/or on a regular basis? Goba

« previous php.mirrors (#16958) next »