Re: Request for opinions: bug vs feature - changeintokenization of yield from

Hi On 7/20/24 20:31, Christoph M. Becker wrote:

If the tokenizer would tokenize a whole file as a single token, would that also be correct? Of course, I'm exaggerating, but

The function is documented as:

token_get_all() parses the given code string into PHP language tokens using the Zend engine's lexical scanner.

So if Zend engine's lexical scanner would tokenize a complete PHP file as a single token, then the output would be correct, yes.

<https://3v4l.org/qIf2c> doesn't look correct to me – "yield /* comment */ from" shouldn't be a single token.

Looking at this from a user expectation perspective and not a technical perspective, neither should yield from be a single token in:

    var_dump(token_get_all('<?php yield from $foo;'));

Best regards Tim Düsterhus