Re: Revert Tokenizer behavior for 5.4

So, is there any consensus about this? Nikita On Fri, Sep 16, 2011 at 10:07 AM, Ferenc Kovacs <tyra3l@gmail.com> wrote: >> >> Wait wait wait. Thats the point here? >> __COMPILER_HALT_OFFSET__ already tells you where the data starts. >> >> -Hannes >> > > I didn't sent this message first, but after reading the mail from > Chris, I think maybe it would clear the confusion: > > It is about tokenizing a file which has __halt_compiler(); in it. > before the fix of the original bugreport, one could get the warning > "Unexpected character in input" if he tried to token_get_all() a > script which had binary data after the __halt_compiler(); > iliaa's fix was trivial: break from the tokenizer if __halt_compiler > token is found. > but it isn't good enough, because as the original bugreporter pointed out: > 1, now the token_get_all() won't return the (); after __halt_compiler, > which means that if you rebuild the code from the tokens, you will > have invalid php code. > 2, you have no way to get the binary data after the __halt_compiler > via the tokenizer, so you can't rebuild the original file using only > the tokenizer. (for example one could use the tokenizer to strip the > whitespaces and comments from a given file in-place) > > both problems could be hacked around from userland, but imo it still > worth fixing those. > > -- > Ferenc Kovács > @Tyr43l - http://tyrael.hu >