Bug #68031 [Com]: htmlspecialchars returns empty string, sometimes

Edit report at https://bugs.php.net/bug.php?id=68031&edit=1 ID: 68031 Comment by: phpbugs at hypertwins dot org Reported by: pfenderd at bellsouth dot net Summary: htmlspecialchars returns empty string, sometimes Status: Open Type: Bug Package: Filter related Operating System: Linux PHP Version: 5.5.16 Block user comment: N Private report: N New Comment: In my case, this does seem to be a character set problem: adding ENT_SUBSTITUTE to the "flags" parameter eliminates the blank results. This appears to be consistent with the documentation for that flag, which says it will "Replace invalid code unit sequences with a Unicode Replacement Character U+FFFD (UTF-8) or &#FFFD; (otherwise) instead of returning an empty string." Apparently, then, returning an empty string is correct behavior in the absence of that flag. Previous Comments: ------------------------------------------------------------------------ [2014-09-17 19:35:02] pfenderd at bellsouth dot net I have been using PHP for 14 years and I have encountered similar problems wth the PHP script iterpreter that gets solved by rearranging the code statements without actually change the statements themselves. This seems to be one of those cases that will never be resolved. Since I found a coding solution that works for me and I cannot recreate the problem in a simple test case, then I guess that we should not waste any more time on this and close the bug report. ------------------------------------------------------------------------ [2014-09-17 19:17:58] rasmus@php.net If you can't reproduce it, how do you know it wasn't a non-utf8 char in the input? What you describe is exactly what htmlspecialchars() does if it encounters and illegal character in the charset it is in. And from 5.3 to 5.4 the default charset changed from iso-8859-1 to UTF8. Those two things combined with the fact that you said it works fine in 5.3 and broke in 5.4 and 5.5 is a lot of evidence that points to the illegal utf-8 char hypothesis. Your hypothesis that it is somehow in the general processor has 0 data points pointing to it other than a really vague one about you moving your code around a bit. ------------------------------------------------------------------------ [2014-09-17 18:59:18] pfenderd at bellsouth dot net As I mentioned in my original post, I got the code working by moving the variable assignment farther down in the code, so I don't think the problem is actually with htmlspecialchars(). I have tried to provide a section of code like the original, but there is no problem with this result. <?php // variable provides mock database search result of actual data that showed the problem. $test_str = <<<XXX Thank you for taking the time to visit us here. We would like to invite you to come and visit our church during any of our services. We are known in the community as a friendly church where the Bible is faithfully taught and preached. Nothing quite compares to the joy of Christian friendship, and at God's First Church of Sample, we make it a priority to build lasting bonds between the members of our church family - bonds of genuine concern and commitment to one another. Best of all, this circle of care is ever widening. We would love to include you as well. We believe that studying the Bible is vital because it not only instructs us intellectually, but it also guides us spiritually. We believe and accept it as God's Word to man, a book that is alive and relevant to life today, and learning its truths can be a thrilling adventure. Opportunities for fellowship and learning are offered to every person at every age level by our staff of qualified teachers and leaders. We have Sunday school classes for children, youth, and adults in which principles from the Bible are taught in an open and personal forum. In addition to our Sunday school classes, we have a discipleship course that helps the new and even the most mature Christian to develop a lifelong, personal, and obedient relationship with Jesus Christ through Biblical teachings. Just as Jesus Christ came, "not to be ministered unto, but to minister..." we accept our responsibility to reach out in service to others. This applies both within the church family and outside our fellowship. Our primary reason for meeting together is to focus our attention on God, giving Him our worship, and receiving His blessing and inspiration. Each time we meet it is a special time of spiritual refreshment. XXX; $cs_info = array( 'church_subdomain_id'=>4, 'cs_subdomain'=>"sample1", 'cs_church'=>"God's First Church of Sample", 'cs_pastor'=>"Sample Simon, Pastor", 'cs_template_no'=>1, 'cs_welcome_title'=>"Welcome to God's First Church of Sample", 'cs_welcome_title2'=> '', 'cs_welcome_msg'=> $test_str, 'cs_address'=>"435 Bridge Ave", 'cs_address2'=>"", 'cs_city'=>"Sampleville", 'cs_state'=>"SC", 'cs_zip'=>"29639" ); $rx = &$cs_info; $church_subdomain_id = $rx['church_subdomain_id']; $cs_subdomain = $rx['cs_subdomain']; $cs_church = htmlspecialchars($rx['cs_church']); $cs_pastor = htmlspecialchars($rx['cs_pastor']); $cs_template_no = $rx['cs_template_no']; $cs_welcome_title = htmlspecialchars($rx['cs_welcome_title']); $cs_welcome_title2 = htmlspecialchars($rx['cs_welcome_title2']); $cs_welcome_msg = htmlspecialchars($rx['cs_welcome_msg']); $xcs_welcome_msg = $rx['cs_welcome_msg']; $cs_address = htmlspecialchars($rx['cs_address']); $cs_address2 = htmlspecialchars($rx['cs_address2']); $cs_city = htmlspecialchars($rx['cs_city']); $cs_state = $rx['cs_state']; $cs_zip = $rx['cs_zip']; // other assignments followed but no others used htmlspecialchars() ?> <!doctype html> <html lang="en"> <head> <title>PHP Bug Test $2</title> </head> <body> <br> Original Variable value (<?php echo $rx['cs_welcome_msg'];?>)<br><br> Value from htmlspecialchars() (<?php echo $cs_welcome_msg;?>)<br><br> </body> </html> ------------------------------------------------------------------------ [2014-09-17 17:58:38] requinix@php.net And you have problems with *that exact script*? We need code that actually fails, not something similar to it; start with the original code you're using and pare it down, removing database requirements and such, until you've reached a fairly minimal version that still breaks. For kicks, I tried running that one a few thousand times (PHP 5.5 on Ubuntu) and every single one worked. ------------------------------------------------------------------------ [2014-09-17 17:45:21] pfenderd at bellsouth dot net <?php $xcs_welcome_msg = <<<XXX Thank you for taking the time to visit us here. We would like to invite you to come and visit our church during any of our services. We are known in the community as a friendly church where the Bible is faithfully taught and preached. Nothing quite compares to the joy of Christian friendship, and at God's First Church of Sample, we make it a priority to build lasting bonds between the members of our church family - bonds of genuine concern and commitment to one another. Best of all, this circle of care is ever widening. We would love to include you as well. We believe that studying the Bible is vital because it not only instructs us intellectually, but it also guides us spiritually. We believe and accept it as God's Word to man, a book that is alive and relevant to life today, and learning its truths can be a thrilling adventure. Opportunities for fellowship and learning are offered to every person at every age level by our staff of qualified teachers and leaders. We have Sunday school classes for children, youth, and adults in which principles from the Bible are taught in an open and personal forum. In addition to our Sunday school classes, we have a discipleship course that helps the new and even the most mature Christian to develop a lifelong, personal, and obedient relationship with Jesus Christ through Biblical teachings. Just as Jesus Christ came, "not to be ministered unto, but to minister..." we accept our responsibility to reach out in service to others. This applies both within the church family and outside our fellowship. Our primary reason for meeting together is to focus our attention on God, giving Him our worship, and receiving His blessing and inspiration. Each time we meet it is a special time of spiritual refreshment. XXX; $cs_welcome_msg = htmlspecialchars($xcs_welcome_msg); ?> <!doctype html> <html lang="en"> <head> <title>PHP Bug Test</title> </head> <body> <br> Original Variable value (<?php echo $xcs_welcome_msg;?>)<br><br> Value from htmlspecialchars() (<?php echo $cs_welcome_msg;?>)<br><br> </body> </html> ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=68031 -- Edit this bug report at https://bugs.php.net/bug.php?id=68031&edit=1