note 91847 added to function.token-get-all

I wanted to use the tokenizer functions to count source lines of code, including counting comments. Attempting to do this with regular expressions does not work well because of situations where /* appears in a string, or other situations. The token_get_all() function makes this task easy by detecting all the comments properly. However, it does not tokenize newline characters. I wrote the below set of functions to also tokenize newline characters as T_NEW_LINE. <?php define('T_NEW_LINE', -1); function token_get_all_nl($source) { $new_tokens = array(); // Get the tokens $tokens = token_get_all($source); // Split newlines into their own tokens foreach ($tokens as $token) { $token_name = is_array($token) ? $token[0] : null; $token_data = is_array($token) ? $token[1] : $token; // Do not split encapsed strings or multiline comments if ($token_name == T_CONSTANT_ENCAPSED_STRING || substr($token_data, 0, 2) == '/*') { $new_tokens[] = array($token_name, $token_data); continue; } // Split the data up by newlines $split_data = preg_split('#(\r\n|\n)#', $token_data, -1, PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY); foreach ($split_data as $data) { if ($data == "\r\n" || $data == "\n") { // This is a new line token $new_tokens[] = array(T_NEW_LINE, $data); } else { // Add the token under the original token name $new_tokens[] = is_array($token) ? array($token_name, $data) : $data; } } } return $new_tokens; } function token_name_nl($token) { if ($token === T_NEW_LINE) { return 'T_NEW_LINE'; } return token_name($token); } ?> Example usage: <?php $tokens = token_get_all_nl(file_get_contents('somecode.php')); foreach ($tokens as $token) { if (is_array($token)) { echo (token_name_nl($token[0]) . ': "' . $token[1] . '"<br />'); } else { echo ('"' . $token . '"<br />'); } } ?> I'm sure you can figure out how to count the lines of code, and lines of comments with these functions. This was a huge improvement on my previous attempt at counting lines of code with regular expressions. I hope this helps someone, as many of the user contributed examples on this website have helped me in the past. ---- Server IP: 72.55.140.6 Probable Submitter: 99.246.63.51 ---- Manual Page -- http://www.php.net/manual/en/function.token-get-all.php Edit -- https://master.php.net/note/edit/91847 Del: integrated -- https://master.php.net/note/delete/91847/integrated Del: useless -- https://master.php.net/note/delete/91847/useless Del: bad code -- https://master.php.net/note/delete/91847/bad+code Del: spam -- https://master.php.net/note/delete/91847/spam Del: non-english -- https://master.php.net/note/delete/91847/non-english Del: in docs -- https://master.php.net/note/delete/91847/in+docs Del: other reasons-- https://master.php.net/note/delete/91847 Reject -- https://master.php.net/note/reject/91847 Search -- https://master.php.net/manage/user-notes.php