Bug #81554 [Asn->Wfx]: RecursiveIteratorIterator still calls ->getChildren() when depth reaches limit
Edit report at https://bugs.php.net/bug.php?id=81554&edit=1
ID: 81554
Updated by: cmb@php.net
Reported by: dktapps at pmmp dot io
Summary: RecursiveIteratorIterator still calls
->getChildren() when depth reaches limit
-Status: Assigned
+Status: Wont fix
Type: Bug
Package: SPL related
Operating System: Windows
PHP Version: 8.0.12
Assigned To: cmb
Block user comment: N
Private report: N
New Comment:
Okay, I'm closing as WONTFIX (for the most part this is not a bug,
but since some edge-cases could be improved â¦)
Previous Comments:
------------------------------------------------------------------------
[2021-10-28 16:01:20] dktapps at pmmp dot io
I guess you're right. I originally discovered this issue in PHPStan, which uses Symfony/Finder
to discover dead files in its cache, which is zero levels deep and only involves files. But I
don't think Symfony uses LEAVES_ONLY for this (some complicated FilterIterator), so I guess it
wouldn't see any benefit anyway.
------------------------------------------------------------------------
[2021-10-28 14:51:16] cmb@php.net
Actually, the inner iterator's ::hasChildren() is called (not
::getChildren)[1]. This is necessary even when the max depth is
reached to determine whether to include the item in the iteration
(if it has no children, and ::LEAVES_ONLY is set) or not.
Unfortunately, RecursiveDirectoryIterator::hasChildren() may cause
up to two stat calls, and these are indeed particularly slow on
Windows.
The only possible optimization I see would be not to call
::hasChildren() if ::LEAVES_ONLY is not set, but that wouldn't
help in your case, and in many other cases since ::LEAVES_ONLY is
the default and likely used most of the time.
Do you agree that this edge-case is not worth optimizing?
[1] <https://github.com/php/php-src/blob/php-7.4.25/ext/spl/spl_iterators.c#L266-L317>
------------------------------------------------------------------------
[2021-10-25 15:39:45] dktapps at pmmp dot io
Description:
------------
When a RecursiveIteratorIterator's depth reaches the limit, it still may call its
sub-iterator's getChildren(). This manifests as performance degradation when using the below
script on a directory with many thousands of files in it.
This can be observed by replacing the iterators with a FilesystemIterator, which by
default won't recurse anyway. As a result, it's 2 orders of magnitude faster than a
RecursiveIteratorIterator with depth 0.
With the target folder containing 30k files (NTFS on a PCIe Gen4 SSD):
- RecursiveDirectoryIterator + maxDepth(0) takes 3.7 seconds
- FilesystemIterator takes 0.03 seconds.
This is most observable on Windows due to Windows' abysmal I/O performance.
Test script:
---------------
Slow script:
<?php
$iterator = new RecursiveDirectoryIterator(sys_get_temp_dir() .
'/phpstan/cache/nette.configurator');
$iterator2 = new RecursiveIteratorIterator($iterator);
$iterator2->setMaxDepth(0);
$start = hrtime(true);
foreach($iterator2 as $item){
}
var_dump(number_format(hrtime(true) - $start));
--------
Fast script:
<?php
$iterator2 = new FilesystemIterator(sys_get_temp_dir() .
'/phpstan/cache/nette.configurator');
$start = hrtime(true);
foreach($iterator2 as $item){
}
var_dump(number_format(hrtime(true) - $start));
Expected result:
----------------
The two scripts should be somewhere in the same order of magnitude of performance.
Actual result:
--------------
The fast script is more than 100x faster than the slow one.
------------------------------------------------------------------------
--
Edit this bug report at https://bugs.php.net/bug.php?id=81554&edit=1
Thread (4 messages)