[Pre-RFC] Lazy iterator_* functions for iterables

From: Date: Sun, 22 Nov 2026 11:26:17 +0000
Subject: [Pre-RFC] Lazy iterator_* functions for iterables
Groups: php.internals 
Request: Send a blank email to internals+get-133181@lists.php.net to get a copy of this message
Hi internals, I'm Paweł, and this is my first post here. Before writing an official RFC, I'd like to gather your feedback on a set of iterator_* functions for iterables. THE PROBLEM Currently, array_* functions only accept arrays. Working with generators, database cursors, or paged APIs requires calling iterator_to_array() first, which has major drawbacks: - It loads the entire sequence into memory. - It prevents early exit (e.g., array_find() or array_any() could stop at the first match). - It fails to terminate on infinite sources. - It silently overwrites duplicate keys by default ($preserve_keys = true). Simply allowing array_* functions to accept iterables wouldn't solve this, as they would still need to consume the entire input to return an array. We need lazy equivalents that return an Iterator. As Rowan noted in the pipe operator thread: "PHP badly needs more native functions for working with iterators" [1]. With the pipe operator (PHP 8.5) and Partial Function Application (PHP 8.6), this gap is even more obvious: chaining operations on an iterable currently means materializing the whole input first, and then building a full intermediate array at every step. PROPOSAL I propose adding 11 functions that mirror existing array_* functions and pass the exact same arguments to their callbacks: - Lazy (returning an Iterator): iterator_map, iterator_filter, iterator_slice, iterator_keys, iterator_values - Terminal (returning a value): iterator_reduce, iterator_any, iterator_all, iterator_find, iterator_find_key, iterator_first For example, with $orders being a generator that reads a file or a database cursor, this reads only as many orders as needed and never holds more than one in memory: $total = $orders |> iterator_filter(?, fn (Order $o) => $o->paid) |> iterator_map(?, fn (Order $o) => $o->amount) |> iterator_slice(?, 0, 100) |> iterator_reduce(?, fn ($sum, $x) => $sum + $x, 0); This set is intentionally closed and introduces no new concepts, leaving all existing array_* and iterator_* functions unchanged. Since the lazy and terminal groups are independent, I plan to split the RFC vote into two parts, just like the array_find RFC did. KEY DECISIONS (full details: [2]) - iterator_ prefix over a namespace: iterator_to_array() and iterator_count() already form a family and have accepted any iterable since PHP 8.2. Per the "Namespaces in bundled extensions" RFC, consistency with existing symbols in an extension takes priority. - Consistent parameter order: the iterable is always first, and the callback is last. This works cleanly with PFA: $x |> iterator_map(?, $fn).. - Key preservation: keys are always preserved (like array_filter()). If you need a re-indexed list, use iterator_values() or iterator_to_array($it, false). This includes iterator_slice(), which, unlike array_slice(), does not renumber integer keys. - Non-rewindable: lazy functions return an Iterator that cannot be rewound once iteration starts, much like a Generator. - No negative offsets/lengths: iterator_slice() rejects negative $offset and $length values, as supporting them would require buffering the entire input in memory. WHY CORE INSTEAD OF USERLAND? Userland libraries like nikic/iter exist, but they differ in naming, parameter order, and namespaces. Adding these to core completes the native iterator_* family and enables zero-dependency pipelines with pipe and PFA. Compared to a PHP polyfill, lazy functions are faster in C (by 16-35% without JIT, 11-33% with JIT), while terminal functions are slower (by 6-10% without JIT, 11-26% with JIT): invoking a PHP callback from C has overhead that the JIT cannot optimize away. This is already true for array_reduce(), which is 3.4x slower than a foreach loop under JIT. The goal isn't raw speed over foreach, but standardization, memory efficiency, and early exit. INTENTIONAL OMISSIONS - No iterator_last(): it would have to consume the entire input and would never terminate on infinite sources. - No extra helpers: functions like chunk, zip, chain, or flat_map are left for future RFCs. - No new interfaces: the only new class is final class LazyIterator, returned by the lazy functions. - Null ambiguity: iterator_find(), iterator_find_key(), and iterator_first() return null when nothing is found, matching array_find(), array_find_key(), and array_first(). iterator_find_key() is additionally ambiguous because, unlike an array key, an iterator key can itself be null. HISTORY The 2021 RFC for any() and all() [3] failed (11:12, with a 2/3 majority required). The most common reason given in its straw poll was that its scope was too small. This proposal offers a broader, yet strictly bounded scope: exact equivalents of existing array_* functions. CURRENT STATUS - Polyfill (PHP 8.2+), .phpt tests, and benchmarks are ready [4]. - Native C implementation in ext/spl with tests is ready [5]. - Benchmark (5M NDJSON lines): iterator_reduce() takes ~2.3 s and 11 KB of memory, whereas iterator_to_array() + array_reduce() takes ~3.9 s and 3.1 GB. For data that is already an array, array_map() and array_filter() remain faster than their lazy counterparts. Disclosure: I wrote the C implementation, polyfill, tests, and benchmarks with the help of an AI assistant (Claude). However, all design decisions are my own. I have thoroughly reviewed all code and verified the tests by intentionally breaking the code they cover. QUESTIONS FOR FEEDBACK 1. Should we stick with the iterator_ prefix, or use a namespace (Iter\ / iter\)? 2. Is this 11-function scope too small, too large, or just right for a single RFC? 3. Is there enough interest to move forward with a formal RFC? If so, could someone please grant wiki RFC karma to my account (pawell67)? Best regards, Paweł Wankiewicz [1] https://externals.io/message/126318#126727 [2] https://github.com/pawell67/iterable-functions#design-decisions [3] https://wiki.php.net/rfc/any_all_on_iterable [4] https://github.com/pawell67/iterable-functions [5] https://github.com/pawell67/php-src/compare/master...iterator-functions

« previous php.internals (#133181) next »