[Pre-RFC] Lazy iterator_* functions for iterables
| From: | Paweł Wankiewicz | Date: | Sun, 22 Nov 2026 11:26:17 +0000 |
| Subject: | [Pre-RFC] Lazy iterator_* functions for iterables | ||
| Groups: | php.internals | ||
| Request: | Send a blank email to internals+get-133181@lists.php.net to get a copy of this message | ||
Hi internals,
I'm Paweł, and this is my first post here. Before writing an official RFC,
I'd like to gather your feedback on a set of iterator_* functions for
iterables.
THE PROBLEM
Currently, array_* functions only accept arrays. Working with generators,
database cursors, or paged APIs requires calling iterator_to_array() first,
which has major drawbacks:
- It loads the entire sequence into memory.
- It prevents early exit (e.g., array_find() or array_any() could stop at
the first match).
- It fails to terminate on infinite sources.
- It silently overwrites duplicate keys by default ($preserve_keys = true).
Simply allowing array_* functions to accept iterables wouldn't solve this,
as they would still need to consume the entire input to return an array. We
need lazy equivalents that return an Iterator. As Rowan noted in the pipe
operator thread: "PHP badly needs more native functions for working with
iterators" [1]. With the pipe operator (PHP 8.5) and Partial Function
Application (PHP 8.6), this gap is even more obvious: chaining operations on
an iterable currently means materializing the whole input first, and then
building a full intermediate array at every step.
PROPOSAL
I propose adding 11 functions that mirror existing array_* functions and
pass the exact same arguments to their callbacks:
- Lazy (returning an Iterator): iterator_map, iterator_filter,
iterator_slice, iterator_keys, iterator_values
- Terminal (returning a value): iterator_reduce, iterator_any, iterator_all,
iterator_find, iterator_find_key, iterator_first
For example, with $orders being a generator that reads a file or a database
cursor, this reads only as many orders as needed and never holds more than
one in memory:
$total = $orders
|> iterator_filter(?, fn (Order $o) => $o->paid)
|> iterator_map(?, fn (Order $o) => $o->amount)
|> iterator_slice(?, 0, 100)
|> iterator_reduce(?, fn ($sum, $x) => $sum + $x, 0);
This set is intentionally closed and introduces no new concepts, leaving all
existing array_* and iterator_* functions unchanged. Since the lazy and
terminal groups are independent, I plan to split the RFC vote into two
parts, just like the array_find RFC did.
KEY DECISIONS (full details: [2])
- iterator_ prefix over a namespace: iterator_to_array() and
iterator_count() already form a family and have accepted any iterable
since PHP 8.2. Per the "Namespaces in bundled extensions" RFC, consistency
with existing symbols in an extension takes priority.
- Consistent parameter order: the iterable is always first, and the
callback is last. This works cleanly with PFA: $x |> iterator_map(?, $fn)..
- Key preservation: keys are always preserved (like array_filter()). If you
need a re-indexed list, use iterator_values() or
iterator_to_array($it, false). This includes iterator_slice(), which,
unlike array_slice(), does not renumber integer keys.
- Non-rewindable: lazy functions return an Iterator that cannot be rewound
once iteration starts, much like a Generator.
- No negative offsets/lengths: iterator_slice() rejects negative $offset
and $length values, as supporting them would require buffering the entire
input in memory.
WHY CORE INSTEAD OF USERLAND?
Userland libraries like nikic/iter exist, but they differ in naming,
parameter order, and namespaces. Adding these to core completes the native
iterator_* family and enables zero-dependency pipelines with pipe and PFA.
Compared to a PHP polyfill, lazy functions are faster in C (by 16-35%
without JIT, 11-33% with JIT), while terminal functions are slower (by
6-10% without JIT, 11-26% with JIT): invoking a PHP callback from C has
overhead that the JIT cannot optimize away. This is already true for
array_reduce(), which is 3.4x slower than a foreach loop under JIT. The goal
isn't raw speed over foreach, but standardization, memory efficiency, and
early exit.
INTENTIONAL OMISSIONS
- No iterator_last(): it would have to consume the entire input and would
never terminate on infinite sources.
- No extra helpers: functions like chunk, zip, chain, or flat_map are left
for future RFCs.
- No new interfaces: the only new class is final class LazyIterator,
returned by the lazy functions.
- Null ambiguity: iterator_find(), iterator_find_key(), and iterator_first()
return null when nothing is found, matching array_find(),
array_find_key(), and array_first(). iterator_find_key() is additionally
ambiguous because, unlike an array key, an iterator key can itself be
null.
HISTORY
The 2021 RFC for any() and all() [3] failed (11:12, with a 2/3 majority
required). The most common reason given in its straw poll was that its
scope was too small. This proposal offers a broader, yet strictly bounded
scope: exact equivalents of existing array_* functions.
CURRENT STATUS
- Polyfill (PHP 8.2+), .phpt tests, and benchmarks are ready [4].
- Native C implementation in ext/spl with tests is ready [5].
- Benchmark (5M NDJSON lines): iterator_reduce() takes ~2.3 s and 11 KB of
memory, whereas iterator_to_array() + array_reduce() takes ~3.9 s and
3.1 GB. For data that is already an array, array_map() and
array_filter() remain faster than their lazy counterparts.
Disclosure: I wrote the C implementation, polyfill, tests, and benchmarks
with the help of an AI assistant (Claude). However, all design decisions are
my own. I have thoroughly reviewed all code and verified the tests by
intentionally breaking the code they cover.
QUESTIONS FOR FEEDBACK
1. Should we stick with the iterator_ prefix, or use a namespace
(Iter\ / iter\)?
2. Is this 11-function scope too small, too large, or just right for a
single RFC?
3. Is there enough interest to move forward with a formal RFC? If so, could
someone please grant wiki RFC karma to my account (pawell67)?
Best regards,
Paweł Wankiewicz
[1] https://externals.io/message/126318#126727
[2] https://github.com/pawell67/iterable-functions#design-decisions
[3] https://wiki.php.net/rfc/any_all_on_iterable
[4] https://github.com/pawell67/iterable-functions
[5] https://github.com/pawell67/php-src/compare/master...iterator-functions