Bug #81554 [Fbk->Asn]: RecursiveIteratorIterator still calls ->getChildren() when depth reaches limit

Edit report at https://bugs.php.net/bug.php?id=81554&edit=1 ID: 81554 User updated by: dktapps at pmmp dot io Reported by: dktapps at pmmp dot io Summary: RecursiveIteratorIterator still calls ->getChildren() when depth reaches limit -Status: Feedback +Status: Assigned Type: Bug Package: SPL related Operating System: Windows PHP Version: 8.0.12 Assigned To: cmb Block user comment: N Private report: N New Comment: I guess you're right. I originally discovered this issue in PHPStan, which uses Symfony/Finder to discover dead files in its cache, which is zero levels deep and only involves files. But I don't think Symfony uses LEAVES_ONLY for this (some complicated FilterIterator), so I guess it wouldn't see any benefit anyway. Previous Comments: ------------------------------------------------------------------------ [2021-10-28 14:51:16] cmb@php.net Actually, the inner iterator's ::hasChildren() is called (not ::getChildren)[1]. This is necessary even when the max depth is reached to determine whether to include the item in the iteration (if it has no children, and ::LEAVES_ONLY is set) or not. Unfortunately, RecursiveDirectoryIterator::hasChildren() may cause up to two stat calls, and these are indeed particularly slow on Windows. The only possible optimization I see would be not to call ::hasChildren() if ::LEAVES_ONLY is not set, but that wouldn't help in your case, and in many other cases since ::LEAVES_ONLY is the default and likely used most of the time. Do you agree that this edge-case is not worth optimizing? [1] <https://github.com/php/php-src/blob/php-7.4.25/ext/spl/spl_iterators.c#L266-L317> ------------------------------------------------------------------------ [2021-10-25 15:39:45] dktapps at pmmp dot io Description: ------------ When a RecursiveIteratorIterator's depth reaches the limit, it still may call its sub-iterator's getChildren(). This manifests as performance degradation when using the below script on a directory with many thousands of files in it. This can be observed by replacing the iterators with a FilesystemIterator, which by default won't recurse anyway. As a result, it's 2 orders of magnitude faster than a RecursiveIteratorIterator with depth 0. With the target folder containing 30k files (NTFS on a PCIe Gen4 SSD): - RecursiveDirectoryIterator + maxDepth(0) takes 3.7 seconds - FilesystemIterator takes 0.03 seconds. This is most observable on Windows due to Windows' abysmal I/O performance. Test script: --------------- Slow script: <?php $iterator = new RecursiveDirectoryIterator(sys_get_temp_dir() . '/phpstan/cache/nette.configurator'); $iterator2 = new RecursiveIteratorIterator($iterator); $iterator2->setMaxDepth(0); $start = hrtime(true); foreach($iterator2 as $item){ } var_dump(number_format(hrtime(true) - $start)); -------- Fast script: <?php $iterator2 = new FilesystemIterator(sys_get_temp_dir() . '/phpstan/cache/nette.configurator'); $start = hrtime(true); foreach($iterator2 as $item){ } var_dump(number_format(hrtime(true) - $start)); Expected result: ---------------- The two scripts should be somewhere in the same order of magnitude of performance. Actual result: -------------- The fast script is more than 100x faster than the slow one. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=81554&edit=1