[php-src] Issue #15287: Pdo\pgSql does not fetch() efficiently

From: Date: Thu, 08 Aug 2024 05:58:50 +0000
Subject: [php-src] Issue #15287: Pdo\pgSql does not fetch() efficiently
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-247462@lists.php.net to get a copy of this message
Issue: https://github.com/php/php-src/issues/15287 Author: outtersg ### Description It would be expected that: ```php <?php $statement = $db->query("select * from that_table"); while(($row = $statement->fetch())) fputcsv(…, $row); ``` be an efficient way to dump a table with PHP being a simple no-store pass-through. But in fact fetch is _not_ lazy as one could hope: **the whole resultset is loaded in the query()** before the script gets a chance to read the first rows. E.g.: on a 24 million entries table of ~ 35 bytes per row, php consumes 2,5 GB of RAM during the whole process (that is, 3 bytes per effective data byte. It could have been worse). In addition to this memory consumption, this prevents maximizing throughput, as we _first_ have to wait the whole network part to finish, _then only_ we do the whole disk I/O. ### Cursors? Here and there I see suggestions to use (server-side) cursors. But: * I'm still looking for how to use the internal ones (cursor_name does not want to get defined) * That's still a lot of overhead on network (and code? Because each call to the cursor is considered a new query, it may have to load column descriptions and so on from scratch) ### Client laziness implementation? [libpq official docs](https://www.postgresql.org/docs/current/libpq-single-row-mode.html) says that PQexec is too high-level for that, and directs to PQsetSingleRowMode after a PQsendQuery. As laziness modifies the script's responsibilities (error handling is probably different), this **should not be the default**. But maybe PDO::ATTR_PREFETCH, used as a boolean (1 being the default to continue using PQexec, so an explicit set to 0 would be required to use PQsetSingleRowMode), could be used to this effect. As for the code structure: the cursors implementation is a great starting point, because its workflow is very similar (necessity to handle interrupts after partial results, to close a previous unclosed cursor…). ### PHP Version PHP 8.3 ### Operating System _No response_

« previous php.bugs (#247462) next »