Re: Manuals and Spiders

I agree, that we should not need a mysql db at each mirror, which is why I mentioned that we could use a db such as cdb / db3 / db4 etc. This does not require any server to be running at the mirror. We can directly use existing php functions to talk to such a database. ( http://www.php.net/manual/en/ref.dba.php ) As the last option we could look at splitting the file. Which is what a database would internally do. So instead of replicating what a db does internally and making our own b-trees out of this file, we could look at using a standard database such as cdb / db3 / db4 as far as possible. I don't have too much prior experience with any of these db's, though I have used cdb files extensively in qmail deployments. And they do provide blazing fast access speeds. If someone on this list has prior exp with these formats, please put in your comments about this. Storing the country code in the cookie is a good idea as it would save us a lookup each time. Though, it may give issues to roaming users. If I am in china right now on a buiness conf, when I first access the php.net website using my laptop, I would have china stored in my cookie for life. So we would have to expire this every 30 days or some other reasonable figure. - Div > -----Original Message----- > From: Gabor Hojtsy [mailto:gabor@hojtsy.hu] > Sent: Wednesday, April 23, 2003 2:08 PM > To: Divyank Turakhia; php-mirrors@lists.php.net > Cc: 'Alex Kiesel'; 'shimi'; 'Jim Winstead'; 'James Cox' > Subject: Re: Manuals and Spiders > > > > > BTW ip-to-country seems to be more open as we would be free to > > > distribute the database via rsync to our mirror sites. If we go on > > > the ip-to-country way however, then we need to find out, whether > > > it is quick enough to search in a single CVS file with PHP, or we > > > need to split the file into segments, and select the target file > > > first based on the IP, and then search for the country info... > > > Speed tests are needed... > > > > Speed should not be an issue, since it is an open database. We can > > actually dump the entire CSV file provided at ip-to-country > into a db3 > > / db3 / cdb / mysql database. Once we properly index the > database we > > should get blazing speeds for each query. > > Having the list in DB is not an option, we don't require any of our > mirrors to have any DB installed, nor do we intend this to be... > > Having this as a web service on rsync.php.net or somewhere > else just puts more load on the rsync server. > > BTW dropping out the quotes and the country names from the > data file, it's size can be reduced to 412Kb, which is more > manageable then the original 731Kb. And country names are not > needed while country codes perfectly identify the countries... > > So as we need to search on the file level, without any > database, the performance question is still up. We probably > need to split up the file. And it may also be a good option > to store the country code in a cookie, after it is found out, > so the code need not run everytime... > > Goba > >