Re: php.nederland.net / aka mirroring php

On Wed, Feb 13, 2002 at 09:50:26AM +0100, Gabor Hojtsy wrote: > OK, but if we bz2 all the manuals, if only one or two page > changes, all the bz2 needs to be downloaded. for the html version, they already do. > I don't know how rsync handles new files (by content or by date, or > some more sophisticated way), but if it handles the files by only > dates, then using this bz2 method would be a significant increase. If > rsync also takes in consideration the content of files (eg producing a > diff), then updating if small changes are made wil be the pain, > instead of updating if no changes are made... "Small changes" occur > more ofthen than "no changes" AFAIK. rsync only sends changed file contents. but i'm not sure that only a few files in the manual change each time it is updated. it looks to me more like all of the individual html and php files from the manual change enough with every manual build because of automatically-generated anchors that they end up getting sent after every build, anyway. some hard numbers: * bytes sent by rsync.php.net when a mirror updates and there are no changes (or with --dry-run): 1,656,580 * bytes sent by rsync.php.net when the same mirror updates with "--exclude='manual/*/html'": 848,844 * bytes saved over 24 hours by excluding manual/*/html, assuming hourly pulls: 19,385,664 * total size of .tar.bz2 html manuals, in bytes: 13,299,712 even if every .tar.bz2 were updated daily, rsync.php.net would be sending less data if the mirrors simply unpacked those into the phpweb tree themselves. (and it would save the disk i/o of having to build that file listing for every mirror update.) i believe that creating a .tar.bz2 of the php-format manual would result in similar savings. it isn't the amount of file data that is making rsync.php.net difficult, it is the number of files that rsync has to send data about so the mirror can check if it has the current files. jim