RE: php.nederland.net / aka mirroring php

So, how hard would it be to implement this? James > -----Original Message----- > From: Jim Winstead [mailto:jimw@apache.org] > Sent: Wednesday, February 13, 2002 6:39 PM > To: Gabor Hojtsy > Cc: php-mirrors@lists.php.net > Subject: Re: php.nederland.net / aka mirroring php > > > On Wed, Feb 13, 2002 at 09:50:26AM +0100, Gabor Hojtsy wrote: > > OK, but if we bz2 all the manuals, if only one or two page > > changes, all the bz2 needs to be downloaded. > > for the html version, they already do. > > > I don't know how rsync handles new files (by content or by date, or > > some more sophisticated way), but if it handles the files by only > > dates, then using this bz2 method would be a significant increase. If > > rsync also takes in consideration the content of files (eg producing a > > diff), then updating if small changes are made wil be the pain, > > instead of updating if no changes are made... "Small changes" occur > > more ofthen than "no changes" AFAIK. > > rsync only sends changed file contents. > > but i'm not sure that only a few files in the manual change each time it > is updated. it looks to me more like all of the individual html and php > files from the manual change enough with every manual build because of > automatically-generated anchors that they end up getting sent after > every build, anyway. > > some hard numbers: > > * bytes sent by rsync.php.net when a mirror updates and there are no > changes (or with --dry-run): 1,656,580 > > * bytes sent by rsync.php.net when the same mirror updates with > "--exclude='manual/*/html'": 848,844 > > * bytes saved over 24 hours by excluding manual/*/html, assuming hourly > pulls: 19,385,664 > > * total size of .tar.bz2 html manuals, in bytes: 13,299,712 > > even if every .tar.bz2 were updated daily, rsync.php.net would be > sending less data if the mirrors simply unpacked those into the phpweb > tree themselves. (and it would save the disk i/o of having to build that > file listing for every mirror update.) > > i believe that creating a .tar.bz2 of the php-format manual would result > in similar savings. > > it isn't the amount of file data that is making rsync.php.net difficult, > it is the number of files that rsync has to send data about so the > mirror can check if it has the current files. > > jim >