Patch version 2: automatic dependency resolution, and new install options --alldeps, plus order-independent uninstall

Hi, As requested, order of packages no longer matters for pear uninstall. To test try this: pear install -a Mail pear uninstall Net_Socket Net_SMTP Mail These are in exactly backwards order, and with the patch, it uninstalls perfectly :) I should note that without this patch, upgrade-all will not work in certain situations because order of upgrading is important, and in some cases, new versions add dependencies. It's not currently in the patch, but it would be possible to add --alldeps and --allreqdeps to upgrade-all as options as well. It would literally involve only adding them to the list of command-line options, as the functionality is independent from command. The question is whether this might be a ridiculous amount of processing/download time, since XML_RPC is used to retrieve the list of releases for every dependency that has not already been processed. Of course, upgrade-all is pretty intensive, so maybe the penalty is worth it to those users? Any opinions? Greg Greg Beaver wrote: > Hi, > > As I mentioned a few days ago in another thread, I've been struggling > with how many dependencies phpDocumentor 2.0 can have on existing PEAR > packages, and how to solve this problem. Well, the solution to one > major stumbling block is attached to this email :). > > The attached patch brings PEAR installation to the same level of > sophistication that apt possesses. For the first time, automatic > dependency resolution and install is possible. It is experimental, but > fully tested. I'd love it if a few brave souls would be willing to back > up their existing pear/PEAR directory and try this patch out, to assist > in the development. I'm also not particularly happy with the length of > the command-line to PEAR_Installer::download() and if anyone can see a > refinement, I've love to get that in here. Note that the patch was made > against the HEAD of php-src/pear, *NOT* version 1.2.1. > > What does the patch do? > ----------------------- > > * makes the order of packages passed to pear install/pear upgrade > unimportant > > In previous versions this would fail: > > pear install HTTP_Request Net_URL > > With this patch, it correctly switches the install order so that > dependencies are installed first. > > * adds the new install options --alldeps and --allreqdeps > > --alldeps instructs pear to recursively find and install all required > and all optional dependencies > --allreqdeps instructs pear to recursively find and install all required > dependencies only > > usage example: pear install --alldeps SOAP > usage example: pear install --a SOAP > > These examples will find the dependencies of SOAP: Mail_Mime, > HTTP_Request, Net_URL, and Net_DIME and download them. > > Then it will find all the dependencies of the dependencies, and download > Net_Socket as a result (Net_URL will not be downloaded twice). If any > of these dependencies are installed and up to date, they will not be > downloaded. If they are installed and not up to date, they will be > upgraded. > > Finally, the packages will be sorted into the order needed to install > them, and installed. I've tested with both Cache (3 levels of > dependency) and SOAP (2 levels plus common dependency across levels) > > > usage example: pear install --allreqdeps > pearcvs/pear/Perm_LiveUser/package.xml > usage example: pear install --A pearcvs/pear/Perm_LiveUser/package.xml > > These examples assume pearcvs/ is a cvs checkout of the pear/ module > from cvs.php.net > > This one will install LiveUser 0.8 from CVS, and skip the download and > install of MDB, DB and HTTP_Request > > * splits PEAR_Installer::install() into 3 methods: private > _downloadFile(), public download() and public install(). This split > allows the download of all files prior to any installation attempt, to > allow full dependency resolution and sorting for installation. > > * Numerous important failure tests > - failure on package not existing, or having no releases yet > - failure if the preferred state is too high for dependencies to be > installed (Net_DIME only has beta releases, for example, so if > preferred_state is stable, it will fail on installation) > > What doesn't the patch do? > -------------------------- > * Major issue: no check for major version upgrade/install - waiting for > resolution on how major upgrade/install will be handled (this is an > existing issue in the pear installer, so I decided not to "fix" it) > > * No change is made to pear uninstall. > > "pear uninstall Net_URL HTTP_Request" will still fail due to incorrect > ordering of packages on the command-line, and "pear uninstall SOAP" will > not attempt to uninstall any dependencies. I didn't feel comfortable > with the safety of adding --alldeps or --allreqdeps to uninstall, but if > there is a demand for it, it can be easily added. (uninstalling > dependencies is MUCH simpler than installing them). > > :) > Greg > > > ------------------------------------------------------------------------ > > Index: pear/PEAR/Common.php > =================================================================== > RCS file: /repository/php-src/pear/PEAR/Common.php,v > retrieving revision 1.100 > diff -u -r1.100 Common.php > --- pear/PEAR/Common.php 13 Aug 2003 21:12:15 -0000 1.100 > +++ pear/PEAR/Common.php 23 Aug 2003 04:45:11 -0000 > @@ -1358,6 +1358,30 @@ > } > > // }}} > + // {{{ betterStates() > + > + /** > + * Return an array containing all of the states that are more stable than > + * or equal to the passed in state > + * > + * @param string Release state > + * @param boolean Determines whether to include $state in the list > + * @return false|array False if $state is not a valid release state > + */ > + function betterStates($state, $include = false) > + { > + static $states = array('devel', 'snapshot', 'alpha', > 'beta', 'stable'); > + $i = array_search($state, $states); > + if ($i === false) { > + return false; > + } > + if ($include) { > + $i--; > + } > + return array_slice($states, $i + 1); > + } > + > + // }}} > // {{{ detectDependencies() > > function detectDependencies($any, $status_callback = null) > @@ -1688,6 +1712,104 @@ > call_user_func($callback, 'done', $bytes); > } > return $dest_file; > + } > + > + // }}} > + // {{{ sortPkgDeps() > + > + /** > + * Sort a list of arrays of array(downloaded packagefilename) by dependency. > + * > + * It also removes duplicate dependencies > + * @param array > + * @return array array of array(packagefilename, package.xml contents) > + */ > + function sortPkgDeps(&$packages) > + { > + foreach($packages as $packagefile) { > + if (!is_array($packagefile)) { > + $ret[] = array('file' => $packagefile, > + 'info' => $a = > $this->infoFromAny($packagefile), > + 'pkg' => $a['package']); > + } else { > + $ret[] = $packagefile; > + } > + } > + $checkdupes = array(); > + $newret = array(); > + foreach($ret as $i => $p) { > + if (!isset($checkdupes[$p['info']['package']])) { > + $checkdupes[$p['info']['package']][] = $i; > + $newret[] = $p; > + } > + } > + > + usort($newret, array('PEAR_Common', '_sortPkgDeps')); > + $packages = $newret; > + } > + > + // }}} > + // {{{ _sortPkgDeps() > + > + /** > + * Compare two package's package.xml, and sort > + * so that dependencies are installed first > + * > + * This is a crude compare, real dependency checking is done on install. > + * The only purpose this serves is to make the command-line > + * order-independent (you can list a dependent package first, and > + * installation occurs in the order required) > + * @access private > + */ > + function _sortPkgDeps($p1, $p2) > + { > + $p1name = $p1['info']['package']; > + $p2name = $p2['info']['package']; > + $p1deps = PEAR_Common::_getPkgDeps($p1); > + $p2deps = PEAR_Common::_getPkgDeps($p2); > + if (!count($p1deps) && !count($p2deps)) { > + return 0; // order makes no difference > + } > + if (!count($p1deps)) { > + return -1; // package 2 has dependencies, package 1 doesn't > + } > + if (!count($p2deps)) { > + return 1; // package 2 has dependencies, package 1 doesn't > + } > + // both have dependencies > + if (in_array($p1name, $p2deps)) { > + return -1; // put package 1 first > + } > + if (in_array($p2name, $p1deps)) { > + return 1; // put package 2 first > + } > + // doesn't really matter if neither depends on the other > + return 0; > + } > + > + // }}} > + // {{{ _getPkgDeps() > + > + /** > + * get an array of package dependency names > + * @access private > + */ > + function _getPkgDeps($p) > + { > + if (!isset($p['info']['releases'])) { > + return array(); > + } > + $rel = array_shift($p['info']['releases']); > + if (!isset($rel['deps'])) { > + return array(); > + } > + $ret = array(); > + foreach($rel['deps'] as $dep) { > + if ($dep['type'] == 'pkg') { > + $ret[] = $dep['name']; > + } > + } > + return $ret; > } > > // }}} > Index: pear/PEAR/Installer.php > =================================================================== > RCS file: /repository/php-src/pear/PEAR/Installer.php,v > retrieving revision 1.98 > diff -u -r1.98 Installer.php > --- pear/PEAR/Installer.php 17 Aug 2003 18:12:43 -0000 1.98 > +++ pear/PEAR/Installer.php 23 Aug 2003 04:45:12 -0000 > @@ -464,26 +464,10 @@ > } > > // }}} > - > - // {{{ install() > - > - /** > - * Installs the files within the package file specified. > - * > - * @param $pkgfile path to the package file > - * > - * @return array package info if successful, null if not > - */ > - > - function install($pkgfile, $options = array()) > + // {{{ _downloadFile() > + function _downloadFile($pkgfile, &$config, $options, &$errors) > { > - // recognized options: > - // - force : force installation > - // - register-only : update registry but don't install files > - // - upgrade : upgrade existing install > - // - soft : fail silently > - // > - $php_dir = $this->config->get('php_dir'); > + $php_dir = $config->get('php_dir'); > if (isset($options['installroot'])) { > if (substr($options['installroot'], -1) == DIRECTORY_SEPARATOR) { > $options['installroot'] = substr($options['installroot'], > 0, -1); > @@ -495,8 +479,6 @@ > } > $this->registry = &new PEAR_Registry($php_dir); > $need_download = false; > - // ==> XXX should be removed later on > - $flag_old_format = false; > if (preg_match('#^(http|ftp)://#', $pkgfile)) { > $need_download = true; > } elseif (!@is_file($pkgfile)) { > @@ -504,22 +486,24 @@ > if ($this->registry->packageExists($pkgfile) && > empty($options['upgrade']) && > empty($options['force'])) > { > - return $this->raiseError("$pkgfile already installed"); > + $errors[] = "$pkgfile already installed"; > + return; > } > $pkgfile = $this->getPackageDownloadUrl($pkgfile); > $need_download = true; > } else { > if (strlen($pkgfile)) { > - return $this->raiseError("Could not open the package file: > $pkgfile"); > + $errors[] = "Could not open the package file: $pkgfile"; > } else { > - return $this->raiseError("No package file given"); > + $errors[] = "No package file given"; > } > + return; > } > } > > // Download package ----------------------------------------------- > if ($need_download) { > - $downloaddir = $this->config->get('download_dir'); > + $downloaddir = $config->get('download_dir'); > if (empty($downloaddir)) { > if (PEAR::isError($downloaddir = System::mktemp('-d'))) { > return $downloaddir; > @@ -533,6 +517,209 @@ > } > $pkgfile = $file; > } > + return $pkgfile; > + } > + > + // }}} > + // {{{ download() > + > + /** > + * Download any files and their dependencies, if necessary > + * > + * @param array a mixed list of package names, local files, or package.xml > + * @param PEAR_Config > + * @param array options from the command line > + * @param array this is the array that will be populated with packages to > + * install. Format of each entry: > + * > + * <code> > + * array('pkg' => 'package_name', 'file' => > '/path/to/local/file', > + * 'info' => array() // parsed package.xml > + * ); > + * </code> > + * @param array this will be populated with any error messages > + * @param string command name (upgrade/install/upgrade-all) > + * @param false private recursion variable > + * @param false private recursion variable > + * @param false private recursion variable > + */ > + function download($packages, &$options, &$config, &$installpackages, > + &$errors, $command, $installed = false, $willinstall = false, $state = false) > + { > + // recognized options: > + // - allreqdeps : install all required dependencies as well > + // - alldeps : install all dependencies, including optional > + // > + if (!$willinstall) { > + $willinstall = array(); > + } > + $mywillinstall = array(); > + > + // download files in this list if necessary > + foreach($packages as $pkgfile) { > + $pkgfile = $this->_downloadFile($pkgfile, $config, $options, $errors); > + if (PEAR::isError($pkgfile)) { > + return $pkgfile; > + } > + $tempinfo = $this->infoFromAny($pkgfile); > + if (isset($options['alldeps']) || > isset($options['allreqdeps'])) { > + // ignore dependencies if there are any errors > + if (!PEAR::isError($tempinfo)) { > + $mywillinstall[strtolower($tempinfo['package'])] = > @$tempinfo['release_deps']; > + } > + } > + $installpackages[] = array('pkg' => $tempinfo['package'], > + 'file' => $pkgfile, 'info' => $tempinfo); > + } > + > + // extract dependencies from downloaded files and then download them > + // if necessary > + if (isset($options['alldeps']) || isset($options['allreqdeps'])) { > + $reg = new PEAR_Registry($config->get('php_dir')); > + if (!$installed) { > + $state = $config->get('preferred_state'); > + $installed = $reg->listPackages(); > + array_walk($installed, create_function('&$v,$k','$v = > strtolower($v);')); > + $installed = array_flip($installed); > + } > + include_once "PEAR/Remote.php"; > + $remote = new PEAR_Remote($config); > + $deppackages = array(); > + // construct the list of dependencies for each file > + foreach ($mywillinstall as $package => $alldeps) { > + if (!is_array($alldeps)) { > + continue; > + } > + foreach($alldeps as $info) { > + if ($info['type'] != 'pkg') { > + continue; > + } > + if (!isset($options['alldeps']) && > isset($info['optional']) && > + $info['optional'] == 'yes') { > + // skip optional deps > + $this->log(0, "skipping Package $package optional dependency > $info[name]"); > + continue; > + } > + // get releases > + $releases = $remote->call('package.info', > $info['name'], 'releases'); > + if (PEAR::isError($releases)) { > + return $releases; > + } > + if (!count($releases)) { > + if (!isset($installed[strtolower($info['name'])])) { > + $errors[] = "Package $package dependency $info[name] ". > + "has no releases"; > + } > + continue; > + } > + $found = false; > + $save = $releases; > + while(count($releases) && !$found) { > + if (!empty($state) && $state != 'any') { > + list($release_version,$release) = each($releases); > + if ($state != $release['state'] && > + !in_array($release['state'], > + $this->betterStates($state))) { > + // drop this release - it ain't stable enough > + array_shift($releases); > + } else { > + $found = true; > + } > + } else { > + $found = true; > + } > + } > + if (!count($releases) && !$found) { > + $get = array(); > + foreach($save as $release) { > + $get = array_merge($get, > + $this->betterStates($release['state'], true)); > + } > + $savestate = array_shift($get); > + $errors[] = "Release for $package dependency $info[name] " . > + "has state '$savestate', requires $state"; > + continue; > + } > + if (in_array(strtolower($info['name']), $willinstall) || > + isset($mywillinstall[strtolower($info['name'])])) { > + // skip upgrade check for packages we will install > + continue; > + } > + if (!isset($installed[strtolower($info['name'])])) { > + // skip upgrade check for packages we don't have installed > + $deppackages[] = $info['name']; > + continue; > + } > + > + // see if a dependency must be upgraded > + $inst_version = $reg->packageInfo($info['name'], > 'version'); > + if (!isset($info['version'])) { > + // this is a rel='has' dependency, check against latest > + if (version_compare($release_version, $inst_version, 'le')) > { > + continue; > + } else { > + $deppackages[] = $info['name']; > + continue; > + } > + } > + if (version_compare($info['version'], $inst_version, > 'le')) { > + // installed version is up-to-date > + continue; > + } > + $deppackages[] = $info['name']; > + } // foreach($alldeps > + } // foreach($willinstall > + > + if (count($deppackages)) { > + // check dependencies' dependencies > + // combine the list of packages to install > + $temppack = array(); > + foreach($installpackages as $p) { > + $temppack[] = strtolower($p['info']['package']); > + } > + foreach($deppackages as $pack) { > + $temppack[] = strtolower($pack); > + } > + $willinstall = array_merge($willinstall, $temppack); > + $this->download($deppackages, $options, $config, $installpackages, > + $errors, $command, $installed, $willinstall, $state); > + } > + } // if --alldeps or --allreqdeps > + } > + > + // }}} > + // {{{ install() > + > + /** > + * Installs the files within the package file specified. > + * > + * @param $pkgfile path to the package file > + * > + * @return array package info if successful, null if not > + */ > + > + function install($pkgfile, $options = array()) > + { > + // recognized options: > + // - force : force installation > + // - register-only : update registry but don't install files > + // - upgrade : upgrade existing install > + // - soft : fail silently > + // > + $php_dir = $this->config->get('php_dir'); > + if (isset($options['installroot'])) { > + if (substr($options['installroot'], -1) == DIRECTORY_SEPARATOR) { > + $options['installroot'] = substr($options['installroot'], > 0, -1); > + } > + $php_dir = $this->_prependPath($php_dir, $options['installroot']); > + $this->installroot = $options['installroot']; > + } else { > + $this->installroot = ''; > + } > + $this->registry = &new PEAR_Registry($php_dir); > + $need_download = false; > + // ==> XXX should be removed later on > + $flag_old_format = false; > > if (substr($pkgfile, -4) == '.xml') { > $descfile = $pkgfile; > @@ -644,19 +831,21 @@ > } > } else { > // checks to do only when upgrading packages > - if (!$this->registry->packageExists($pkgname)) { > +/* if (!$this->registry->packageExists($pkgname)) { > return $this->raiseError("$pkgname not installed"); > - } > - $v1 = $this->registry->packageInfo($pkgname, 'version'); > - $v2 = $pkginfo['version']; > - $cmp = version_compare("$v1", "$v2", 'gt'); > - if (empty($options['force']) && > !version_compare("$v2", "$v1", 'gt')) { > - return $this->raiseError("upgrade to a newer version ($v2 is not newer > than $v1)"); > - } > - if (empty($options['register-only'])) { > - // when upgrading, remove old release's files first: > - if (PEAR::isError($err = $this->_deletePackageFiles($pkgname))) { > - return $this->raiseError($err); > + }*/ > + if ($this->registry->packageExists($pkgname)) { > + $v1 = $this->registry->packageInfo($pkgname, 'version'); > + $v2 = $pkginfo['version']; > + $cmp = version_compare("$v1", "$v2", 'gt'); > + if (empty($options['force']) && > !version_compare("$v2", "$v1", 'gt')) { > + return $this->raiseError("upgrade to a newer version ($v2 is not > newer than $v1)"); > + } > + if (empty($options['register-only'])) { > + // when upgrading, remove old release's files first: > + if (PEAR::isError($err = $this->_deletePackageFiles($pkgname))) { > + return $this->raiseError($err); > + } > } > } > } > @@ -743,7 +932,7 @@ > } > > // Register that the package is installed ----------------------- > - if (empty($options['upgrade'])) { > +/* if (empty($options['upgrade'])) { > // if 'force' is used, replace the info in registry > if (!empty($options['force']) && > $this->registry->packageExists($pkgname)) { > $this->registry->deletePackage($pkgname); > @@ -751,7 +940,18 @@ > $ret = $this->registry->addPackage($pkgname, $pkginfo); > } else { > $ret = $this->registry->updatePackage($pkgname, $pkginfo, false); > + }*/ > + // if 'force' is used, replace the info in registry > + $pkg_installed = $this->registry->packageExists($pkgname); > + if (!empty($options['force']) && $pkg_installed) { > + $this->registry->deletePackage($pkgname); > } > + if (!$pkg_installed) { > + $ret = $this->registry->addPackage($pkgname, $pkginfo); > + } else { > + $ret = $this->registry->updatePackage($pkgname, $pkginfo, false); > + } > + > if (!$ret) { > return null; > } > Index: pear/PEAR/Command/Install.php > =================================================================== > RCS file: /repository/php-src/pear/PEAR/Command/Install.php,v > retrieving revision 1.47 > diff -u -r1.47 Install.php > --- pear/PEAR/Command/Install.php 5 Aug 2003 15:36:07 -0000 1.47 > +++ pear/PEAR/Command/Install.php 23 Aug 2003 04:45:13 -0000 > @@ -69,6 +69,14 @@ > 'ignore-errors' => array( > 'doc' => 'force install even if there were errors', > ), > + 'alldeps' => array( > + 'shortopt' => 'a', > + 'doc' => 'install all required and optional > dependencies', > + ), > + 'allreqdeps' => array( > + 'shortopt' => 'A', > + 'doc' => 'install all required dependencies', > + ), > ), > 'doc' => '<package> ... > Installs one or more PEAR packages. You can specify a package to > @@ -273,9 +281,23 @@ > $this->ui->outputData(array('data' => "Will upgrade > $package"), $command); > } > } > - foreach ($params as $pkg) { > - $bn = basename($pkg); > - $info = $this->installer->install($pkg, $options, $this->config); > + $errors = array(); > + $downloaded = array(); > + if (isset($options['alldeps']) || isset($options['allreqdeps'])) { > + $options['upgrade'] = true; > + } > + $this->installer->download($params, $options, $this->config, $downloaded, > + $errors, $command); > + if (count($errors)) { > + $err['data'] = array($errors); > + $err['headline'] = 'Install Errors'; > + $this->ui->outputData($err); > + return $this->raiseError("$command failed"); > + } > + $this->installer->sortPkgDeps($downloaded); > + foreach ($downloaded as $pkg) { > + $bn = basename($pkg['file']); > + $info = $this->installer->install($pkg['file'], $options, > $this->config); > if (is_array($info)) { > if ($this->config->get('verbose') > 0) { > $label = "$info[package] $info[version]";