Doc #51056 [ReO]: fread() on blocking stream will block even if data is available

Edit report at http://bugs.php.net/bug.php?id=51056&edit=1 ID: 51056 Updated by: cataphract@php.net Reported by: magicaltux@php.net Summary: fread() on blocking stream will block even if data is available Status: Re-Opened Type: Documentation Problem Package: Streams related Operating System: Linux Gentoo 2.6.32 PHP Version: 5.3.1 Block user comment: N New Comment: A small correction: it's not that "never reads from the socket more than one packet at a time" as I and the manual say. It's that it does only one call to recv(). If we're blocking waiting for data and a packet arrives, then recv() will return only the contents of that packet ("The receive calls normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested."). However, if several packets have been received since the last call to fread, recv() will return the most data it can, possibly several packets. But this is a minor documentation issue and not very relevant in this discussion. Previous Comments: ------------------------------------------------------------------------ [2010-09-28 01:55:15] cataphract@php.net The bug reported in right on this issue. As it stands, it is completely unpredictable whether an fread call will block. The point of putting a stream into the readfs set of stream_select is to know whether a call to fread will block or not. stream_select has an emultaion feature that returns the stream if the stream buffer has data, even if there's no more data to be read on the socket. See bug #52602. As it stands, if there's data in the stream buffer but not on the socket and the user asks for more data than what's in the buffer, the call to fread will block, even though stream_select returned the stream. The usual (and documented) semantics of select() would mean a call wouldn't block: «The streams listed in the read array will be watched to see if characters become available for reading (more precisely, to see if a read will not block - in particular, a stream resource is also ready on end-of-file, in which case an fread() will return a zero length string).» Therefore, it's the current behavior that's unpredictable. We can never know whether a call to fread will block. The solution could be: * Return whatever there's on the buffer, even if it's less than what was asked. * Try to fill the rest of the buffer only with non-blocking reads. * Call the real select (no emulation like in stream_select) and fill the rest of the buffer only with the pending data. It would be unwise, however, to have this behavior change introduced for non-sockets. The problem is that many scripts admit that if they asked for n bytes, they will receive n bytes, except in the last call. For sockets, where the problem is more serious (since we never know when a packet will arrive, we can block for a long time), this would be a minor BC break, because in that case fread never reads from the socket more than one packet at a time and since the application can't know whether the data it expects will arrive on one or several packets it already has to do deal with variable-length fread returns. Indeed, the documentation for fread says: «When reading from anything that is not a regular local file, such as streams returned when reading remote files or from popen() and fsockopen(), reading will stop after a packet is available. This means that you should collect the data together in chunks as shown in the examples below.» ------------------------------------------------------------------------ [2010-03-12 14:53:55] lbarnaud@php.net I see your point in wanting read() behavior. Whether or not to implement fread() or read() one is arguable. However the specific behavior you are asking for is not reliable for several reasons, and IMHO (I may be wrong) you want this behavior for bad reasons. Let me explain this : > By the way using nonblocking mode makes no sense with provided example. It would just make the program use 100% cpu. This is why you don't want to use non-blocking streams. If you use stream_select() you will never end up using 100% CPU : Your PHP process will only do an idle wait in stream_select() and consume no CPU at all. Example : stream_set_blocking($stream, 0); while (stream_select($r,$w,$e, $stream, $sec, $usec)) { /* block until data is available for read and/or write in $stream. */ $data = fread($stream, 8192); /* read all available data, up to 8192 bytes. Returns only 1 byte if only 1 byte is available and never blocks. */ } > If end of email is reached while a read is in progress and a new read is called, it will block until the server closes connections With your patch (or with the read behavior you want) it will still block. And it will block randomly, in an unpredictable manner. Please see the following example : Say the buffer has 250 bytes in it. fread(100) -> buffer.length-=100, buffer.length == 100 fread(100) -> buffer.length-=100, buffer.length == 50 fread(100) -> with your patch it would return the last 50 available bytes Now this other example with a buffer with only 200 bytes in it : Say the buffer has 200 bytes in it. fread(100) -> buffer.length-=100, buffer.length == 100 fread(100) -> buffer.mength-=100, buffer.length == 0 fread(100) -> buffer is 0, this blocks, and you can't control this (you don't control the buffer, and don't know anything about it in a php script) Please see 51056-3.phpt. With current behavior it will block too, but in a predictable maner. ------------------------------------------------------------------------ [2010-03-12 03:12:35] magicaltux@php.net So, it is normal for php's fread() to return immediatly when less data than asked is available, unless this data arrived while a previous call of fread() was done and there was too much data ? Let me just state that this doesn't makes sense. I tested stdc's fread() and could confirm that its behaviour is consistent: it will only return when it has collected the data it needed, when EOF is reached or when an error occurs. It seems that PHP's php_stream_read() is closer to read() syscall than to stdc's fread(), except for this one specific behaviour. > It follows fread() behavior since years and I believe it should not change. I believe the problem comes from the new streams api which is an attempt to make the socket api obsolete. In fact stream functions (including fread()) behave the same way the old socket counterpart did when passed a socket. The correct behaviour (as defined by common sense, and confirmed by PHP 4.4.9) : Testing PHP version: 4.4.9 socket_read took 0.06ms to read 8 bytes socket_read took 5.08ms to read 256 bytes socket_read took 0.01ms to read 45 bytes socket_read took 0.08ms to read 8 bytes socket_read took 5.06ms to read 256 bytes socket_read took 0.01ms to read 45 bytes socket_read took 0.07ms to read 8 bytes socket_read took 5.05ms to read 256 bytes socket_read took 0.01ms to read 45 bytes socket_read took 0.08ms to read 8 bytes Testing with PHP 5.1.0 (first version containing stream_socket_pair()) exhibits a change of behaviour due to the new stream api. Both tests 51056.phpt and 51056-2.phpt pass on PHP 4.4.9. By the way using nonblocking mode makes no sense with provided example. It would just make the program use 100% cpu. For example a PHP program reading an email from a POP3 server might lockdown because of this bug in blocking mode. If end of email is reached while a read is in progress and a new read is called, it will block until the server closes connections (expected behaviour = return remaining data). As a PHP sockets programmer (I believe my experience when it comes to php and sockets is not negligeable) I say once more that *this* fread()'s behaviour is not consistent. fread() in blocking mode should block until it has enough bytes or return as soon as some bytes are avaialble. Blocking should not depend on when data has arrived. ------------------------------------------------------------------------ [2010-03-11 22:03:51] lbarnaud@php.net > I still believe fread() should not hang when it has data it can return. It follows fread() behavior since years and I believe it should not change. > The C counterpart doesn't C's fread() does :) > and the manual says it doesn't. The manual looks wrong on this point, "reading will stop after a packet is available" is never true, whatever packet means. fread() (both PHP's and C's) returns less data than asked only on EOF or errors. The only reliable way of doing non-blocking i/o is still to use non-blocking streams ;-) ------------------------------------------------------------------------ [2010-03-11 21:39:48] magicaltux@php.net I still believe fread() should not hang when it has data it can return. The C counterpart doesn't, and the manual says it doesn't. Regarding test 51056-2.phpt.txt the manual explicitly says that this *can happen* on anything else than files (read warning in example #3 on http://php.net/fread ) While I understand your concern for people who might be relying on current bogus behaviour I find this very unlikely considering network streams are subject to lags and different kinds of behaviour due to the large amount of tcp implementations on internet. In the worst case, the manual explicitly warns against relying on fread() returning as many bytes as requested, and says buffering must be used. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/bug.php?id=51056 -- Edit this bug report at http://bugs.php.net/bug.php?id=51056&edit=1