Req #63671 [Opn]: Create a mb_str_word_count() function which is multi-byte aware

Edit report at https://bugs.php.net/bug.php?id=63671&edit=1 ID: 63671 Updated by: cmb@php.net Reported by: kobrien at kiva dot org Summary: Create a mb_str_word_count() function which is multi-byte aware Status: Open Type: Feature/Change Request Package: mbstring related Operating System: Ubuntu 12.04 PHP Version: 5.5.0alpha1 Block user comment: N Private report: N New Comment: It occurs to me that an mb_str_word_count() should not use iswalpha(), because the latter is depending on the current system locale, but mbstring usually doesn't. That doesn't mean that a locale aware str_word_count() wouldn't be useful, but one could easily implement an mb_str_word_count() in userland. Simplified and non-optimized: function mb_str_word_count($string) { return count(mb_split('[\s_"], $string)); } Previous Comments: ------------------------------------------------------------------------ [2012-12-03 03:09:53] kobrien at kiva dot org Description: ------------ Create a mb_str_word_count() function which will properly handle counting the number of words in string that contains multi-byte characters. This is currently not possible with str_word_count() because of use of the isalpha() C function which does not properly handle multi-byte characters. As suggested by aharvey, this new function would replace usage of isalpha() with iswalpha(). A naive (meaning no real knowledge of this or testing of it) patch would look like: diff --git a/ext/standard/string.c b/ext/standard/string.c index 7a4ae2e..9ab6b5f 100644 --- a/ext/standard/string.c +++ b/ext/standard/string.c @@ -5202,7 +5202,7 @@ PHP_FUNCTION(str_word_count) while (p < e) { s = p; - while (p < e && (isalpha((unsigned char)*p) || (char_list && ch[(unsigned char)*p]) || *p == '\'' || *p == '-')) { + while (p < e && (iswalpha((unsigned char)*p) || (char_list && ch[(unsigned char)*p]) || *p == '\'' || *p == '-')) { p++; } if (p > s) { Test script: --------------- <?php // existing str_word_count function for comparison print str_word_count("PHP function str_word_count does not properly handle non-latin characters") . "\n"; // returns 11 print str_word_count("Хабилло житель Яванского района. Ему 70 лет. Он женат. У него четверо детей. Хабилло филолог. Он более двадцати лет работает по профессии. Также Хабилло занимается виноградарством. У него имеется небольшой виноградник. Этим видом деятельности Хабилло занимается 15 лет."); // returns 0 // new function mb_str_word_count print mb_str_word_count("Хабилло житель Яванского района. Ему 70 лет. Он женат. У него четверо детей. Хабилло филолог. Он более двадцати лет работает по профессии. Также Хабилло занимается виноградарством. У него имеется небольшой виноградник. Этим видом деятельности Хабилло занимается 15 лет."); // returns 37 Expected result: ---------------- Using mb_str_word_count() will return the number of words in a string containing multibyte characters Actual result: -------------- Currently there is no mb_str_word_count() function. Using str_word_count() on a string with multibyte characters returns 0. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=63671&edit=1