RE: Manuals and Spiders
| From: | Divyank Turakhia | Date: | Wed, 23 Apr 2003 17:41:48 +0000 |
| Subject: | RE: Manuals and Spiders | ||
| References: | 1 | Groups: | php.mirrors |
| Request: | Send a blank email to php-mirrors+get-16957@lists.php.net to get a copy of this message | ||
> I think it would be doable to request the database from your
> server from
> time to time with a cronned PHP script. Then it can do any
> "fixing" in
> case you won't provide us exactly what is suitable...
I think a cron is a good idea - at least to start off this project. I
can do the fixing at our end, you just need to define the exact format
that you want. What we will do, is give you a url like
http://www.directi.com/iptocountry.php?site=php.net&dbformat=1&lastupdat
e=unixtimestamp to update the db. This will give you the entire db in
the exact format that you want. This way the fixing code does not need
to be there on every mirror. The db is fixed as per php.net's
requirements on our server itself. Also, if we ever update the db in a
different format - in case we start giving more details than what exists
in the current db - we would just need to update the script at a single
location. This will ensure that php and all its related sites would
always get the db in the correct format.
> Every piece of our current system works with three letter
> country codes
> for example (mirror images, mirror country specs). So IMHO it
> would be
> the best to have this list with three letter codes, instead of two
> letter ones.
>
> Staying with the textfile based model, it would probably be
> the best to
> have fixed width fields (IP numbers padded with zeros at the
> left), so
> we can build up a small index to help seek in the file. Then
> there won't
> be any need to have more then one data file. The index would help us
> seek near to the place where the IP info is stored, so we
> only need to
> check some IP ranges there, near to the IP...
>
> This may lead to a significantly bigger file because of the padding
> zeros and the three letter country codes, but we can leave out the
> commas in this case (and the country name is definitely not
> needed as we
> have a mapping array for this:
> http://www.php.net/include/countries.inc )
>
Just give me a sample of the format that you want - by changing 2-3
lines of the current db as per the new format and sending it across to
me / on the list. Once I have this, we can easily write a script to
convert the current db format to the one you want. We will then provide
you with the url that you can put in the cron to get the db
automatically.
The params to be included in the update url are as below. Let me know if
any one can think of any more essential params, or if anyone thinks a
particular param should not exist:
http://www.directi.com/iptocountry.php?site=php.net&dbformat=1&lastupdat
e=unixtimestamp
Site - would tell us which group of sites is requesting this data. So
all php related websites would send this as php.net
Dbformat - in case we start giving additional params that you want to
use, we just need to update the version number. This will allow us to
maintain a single script for all formats. We will provide the update as
per the format requested.
Lastupdate - we can return a "no update", in case there is no updated db
since the last update date.
The cron on the server should call a local php script which will make a
call to this URL. In case php decides to change the dbformat then this
local script is updated using rsync. The cron does not need to change
for ever.
- Divyank