Re: Revert Tokenizer behavior for 5.4

> > Wait wait wait. Thats the point here? > __COMPILER_HALT_OFFSET__ already tells you where the data starts. > > -Hannes > I didn't sent this message first, but after reading the mail from Chris, I think maybe it would clear the confusion: It is about tokenizing a file which has __halt_compiler(); in it. before the fix of the original bugreport, one could get the warning "Unexpected character in input" if he tried to token_get_all() a script which had binary data after the __halt_compiler(); iliaa's fix was trivial: break from the tokenizer if __halt_compiler token is found. but it isn't good enough, because as the original bugreporter pointed out: 1, now the token_get_all() won't return the (); after __halt_compiler, which means that if you rebuild the code from the tokens, you will have invalid php code. 2, you have no way to get the binary data after the __halt_compiler via the tokenizer, so you can't rebuild the original file using only the tokenizer. (for example one could use the tokenizer to strip the whitespaces and comments from a given file in-place) both problems could be hacked around from userland, but imo it still worth fixing those. -- Ferenc Kovács @Tyr43l - http://tyrael.hu