Bug #16939 Updated: tokenizer: emalloc causes segfault

ID: 16939 Updated by: andrei@php.net Reported By: alan@akbkhome.com -Status: Open +Status: Closed Bug Type: Reproducible crash Operating System: linux PHP Version: 4.2.0 New Comment: This bug has been fixed in CVS. You can grab a snapshot of the CVS version at http://snaps.php.net/. In case this was a documentation problem, the fix will show up soon at http://www.php.net/manual/. In case this was a PHP.net website problem, the change will show up on the PHP.net site and on the mirror sites. Thank you for the report, and for helping us make PHP better. Previous Comments: ------------------------------------------------------------------------ [2002-05-04 01:06:00] alan@akbkhome.com OK, Got a nice simple test case - I'm pretty certain its the tokenizer code, Andrei :) -- theres not category for tokenizer yet! :) test script as follows (modify the directory to point at pear).. <? dl('tokenizer.so'); class test { function testing($f) { if (!preg_match('/\.(php|class)$/',$f)) return; echo "$f\n"; $this->data =& token_get_all($f); } function recurse($dir) { $dh = opendir($dir); while (($f = readdir($dh)) !== FALSE) { if ($f{0} == ".") continue; $fn = "{$dir}/{$f}"; if (is_dir($fn)) { $this->recurse($fn); continue; } // assume its a file $this->testing($fn); } } } $t = new test; // change this to your pear directory. $t->recurse('/usr/lib/php'); ?> ------------------------------------------------------------------------ [2002-05-01 03:09:54] alan@akbkhome.com Workaround: Original code did this $this->tokens = token_get_all($filename) by not reassigning the same variable on each file, but rather really storing the token array in a huge array it works around this issue... (albiet with a huge memory overhead) $this->all_tokens[$filename] = token_get_all($contents); $this->tokens =& $this->all_tokens[$filename]; ------------------------------------------------------------------------ [2002-05-01 02:40:23] alan@akbkhome.com A short script to reproduce this (sorry it not to easy to reproduce without the exact code, I did at one point get to the situation where adding a extra space in the white space area solved this!), but I suspect it is that the machine is running low on memory or something.... http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/phpmole-ide/phpmole/tools/uml_generator.class?rev=1.2&content-type=text/vnd.viewcvs-markup http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/phpmole-ide/phpmole/tools/uml_dia.xml.php?rev=1.1&content-type=text/vnd.viewcvs-markup -- just use it do do a large number of files. eg. php uml_generator.class /usr/local/php/pear/ Possible reasons that I could think of: -the tokenizer extension (which may be leaking memory) -somewhere not exiting if no memory can be allocated and then emalloc / malloc being called again. Configuration on this machine (4.2.0RC2), also reproduced with 4.2 release version './configure' '--prefix=/usr' '--without-mysql' '--with-config-file-path=/etc/php4/cgi/' '--enable-dba' '--with-db2' and tokenizer as a dl'd module Backtrace Program received signal SIGSEGV, Segmentation fault. 0x4014db2b in malloc () from /lib/libc.so.6 (gdb) bt #0 0x4014db2b in malloc () from /lib/libc.so.6 #1 0x4014d1e4 in malloc () from /lib/libc.so.6 #2 0x80c4c5c in _emalloc (size=35) at zend_alloc.c:165 #3 0x80d4b1a in zend_hash_index_update_or_next_insert (ht=0x817ae9c, h=207, pData=0xbfffda28, nDataSize=4, pDest=0x0, flag=4) at zend_hash.c:404 #4 0x80d303a in add_next_index_stringl (arg=0x8326d64, str=0x8268c08 ")", length=1, duplicate=1) at zend_API.c:847 #5 0x401ffc61 in tokenize () from /usr/lib/php4/cgi/tokenizer.so #6 0x402005ac in zif_token_get_all () from /usr/lib/php4/cgi/tokenizer.so #7 0x80eb4b0 in execute (op_array=0x816631c) at ./zend_execute.c:1598 #8 0x80eb679 in execute (op_array=0x8166a84) at ./zend_execute.c:1638 #9 0x80eb679 in execute (op_array=0x816004c) at ./zend_execute.c:1638 #10 0x80d19e8 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at zend.c:810 #11 0x805e1dd in php_execute_script (primary_file=0xbffffb98) at main.c:1381 #12 0x805c0ac in main (argc=3, argv=0xbffffc14) at cgi_main.c:785 #13 0x400f86cf in __libc_start_main () from /lib/libc.so.6 ------------------------------------------------------------------------ -- Edit this bug report at http://bugs.php.net/?id=16939&edit=1