Bug #63732 [Com]: unicode strings not handled correctly
Edit report at https://bugs.php.net/bug.php?id=63732&edit=1
ID: 63732
Comment by: jmichae3 at yahoo dot com
Reported by: jmichae3 at yahoo dot com
Summary: unicode strings not handled correctly
Status: Not a bug
Type: Bug
Package: Scripting Engine problem
Operating System: linux
PHP Version: 5.3.19
Block user comment: N
Private report: N
New Comment:
something is wrong with the code you gave me. it doesn't work.
also, it doesn't follow Ken Thompson's table shown below in URL in comment.
there are any other reference documents (wikipedia).
one thing I discovered about this code is that:
- the page must be encoded as UTF-8 without BOM (such as using notepad++, Encoding, Convert to utf8
without BOM.
- you must also include a meta tag
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
better coded utf-8 only version and one that actually works:
(and by the way, this works with ascii too)
//returns ordinal value of character in string $str at $index
//and increments $index past current utf-8 character.
//based on the table at http://doc.cat-v.org/bell_labs/utf-8_history
function utf8_ord_next_char($str, &$index) {
if ($index+1 <= strlen($str)
&& 0x80 == 0x80 & ord($str[$index + 0])
) {
$result =
(ord($str[$index+0])&0x7f) ;
$index += 1;
return $result;
} else if ($index+2 <= strlen($str)
&& 0xc0 == 0xe0 & ord($str[$index + 0])
&& 0x80 == 0xc0 & ord($str[$index + 1])
) {
$result =
(ord($str[$index+0])&0x1f) +
(ord($str[$index+1])&0x3f) ;
$index += 2;
return $result;
} else if ($index+3 <= strlen($str)
&& 0xe0 == 0xf0 & ord($str[$index + 0])) {
&& 0x80 == 0xc0 & ord($str[$index + 1])
&& 0x80 == 0xc0 & ord($str[$index + 2])
) {
$result =
(ord($str[$index+0])&0x0f) +
(ord($str[$index+1])&0x3f) +
(ord($str[$index+2])&0x3f) ;
$index += 3;
return $result;
} else if ($index+4 <= strlen($str)
&& 0xf0 == 0xf8 & ord($str[$index + 0])) {
&& 0x80 == 0xc0 & ord($str[$index + 1])
&& 0x80 == 0xc0 & ord($str[$index + 2])
&& 0x80 == 0xc0 & ord($str[$index + 3])
) {
$result =
(ord($str[$index+0])&0x07) +
(ord($str[$index+1])&0x3f) +
(ord($str[$index+2])&0x3f) +
(ord($str[$index+3])&0x3f) ;
$index += 4;
return $result;
} else if ($index+5 <= strlen($str)
&& 0xf8 == 0xfc & ord($str[$index + 0])) {
&& 0x80 == 0xc0 & ord($str[$index + 1])
&& 0x80 == 0xc0 & ord($str[$index + 2])
&& 0x80 == 0xc0 & ord($str[$index + 3])
&& 0x80 == 0xc0 & ord($str[$index + 4])
) {
$result =
(ord($str[$index+0])&0x03) +
(ord($str[$index+1])&0x3f) +
(ord($str[$index+2])&0x3f) +
(ord($str[$index+3])&0x3f) +
(ord($str[$index+4])&0x3f) ;
$index += 5;
return $result;
} else if ($index+6 <= strlen($str)
&& 0xfc == 0xfe & ord($str[$index + 0])) {
&& 0x80 == 0xc0 & ord($str[$index + 1])
&& 0x80 == 0xc0 & ord($str[$index + 2])
&& 0x80 == 0xc0 & ord($str[$index + 3])
&& 0x80 == 0xc0 & ord($str[$index + 4])
&& 0x80 == 0xc0 & ord($str[$index + 5])
) {
$result =
(ord($str[$index+0])&0x01) +
(ord($str[$index+1])&0x3f) +
(ord($str[$index+2])&0x3f) +
(ord($str[$index+3])&0x3f) +
(ord($str[$index+4])&0x3f) +
(ord($str[$index+5])&0x3f) ;
$index += 6;
return $result;
}
//unknown case
$result = ord($str[$index+0]);
$index++;
return $result;
}
Previous Comments:
------------------------------------------------------------------------
[2012-12-12 04:49:07] rasmus@php.net
You have just slightly reformatted the code I gave you.
------------------------------------------------------------------------
[2012-12-12 04:36:19] jmichae3 at yahoo dot com
this code might be moreuseful, I am going to give it to you.
I know there is unicode-16 and unicode-32 and such.
if the string can hanbdle stuff like that, there really should be an internal function for that
which also handles this internally. because although this is useful and I can use it, it is a
workaround rather than a real and complete solution for multiple encodings such as you would find
listed with mb_list_encodings().
//returns ordinal value of character in string $str at $index
//and increments $index past current utf-8 character.
function utf8_ord_next_char($str, &$index) {
$b0 = ord($str[$index + 0]);
if ($b0 < 0x10) {
$index++;
return $b0;
}
$b1 = ord($str[$index + 1]);
if ($b0 < 0xE0) {
$index += 2;
return (($b0 & 0x1F) << 6) + ($b1 & 0x3F);
}
$index += 3;
return (($b0 & 0x0F) << 12) + (($b1 & 0x3F) << 6) + (ord($str[$index + 2])
& 0x3F);
}
so for detecting non-ascii languages,
//detect foreign languages
for ($i=0;$i < strlen($comment);) {
if (utf8_ord_next_char($comment,$i) > 126) {
echo "<div style='color:red;'>ERRORb</div>";
return true; //error
}
}
------------------------------------------------------------------------
[2012-12-12 02:38:50] rasmus@php.net
Personally I'd just convert from utf8 to iso-8959-1 or whichever encoding you
are looking for here instead of checking each character. But if you really do
want to do it, it isn't very hard. You just need to understand what UTF-8 looks
like and it becomes a simple 5-line function in userspace:
function utf8_ord($c) {
$b0 = ord($c[0]);
if($b0 < 0x10) return $b0;
$b1 = ord($c[1]);
if($b0 < 0xE0 )return (($b0 & 0x1F) << 6) + ($b1 & 0x3F);
return (($b0 & 0x0F) << 12) + (($b1 & 0x3F) << 6) + (ord($c[2]) & 0x3F);
}
But you have to understand that there is absolutely no way to accurately detect
the encoding of a short sequence of bytes. The above will work if you know the
input is UTF-8. There is no way to write a magic function which will tell you
the encoding from a couple of bytes of data which you seem to imply we should
provide you.
------------------------------------------------------------------------
[2012-12-12 00:34:00] jmichae3 at yahoo dot com
if you were to take the time to do the research, there is no function in PHP except ord() for
converting a character [from a string] to a number. maybe strings need to be handled differently
internally in php to handle UNICODE. or maybe ord simply needs to be rewritten so it works so matter
what character encoding is thrown at it. it would be difficult, but extremely useful, since it is
the only function. I took the time to look through the mb functions. there was nothing to help me.
I tried looking through the mb functions, there wasn't a compare. there wasn't a way to
compare. I consider a function like that to be crucial if relops are not safe or capable of doing
it. if that is the case, please make one, and an mb function for returning the ordinal value of an
mb char. the functionality is just not there. thanks. much appreciated.
unicode/mb-related bug database stuff:
https://bugs.php.net/bug.php?id=49439
https://bugs.php.net/bug.php?id=63732
just search the database for anything with mb_encode or unicode. there are a number of bugs related
to this problem.
------------------------------------------------------------------------
[2012-12-11 22:22:24] rasmus@php.net
This is a bug reporting system. You reported a bug on a function that is behaving
as intended and as documented. This is not a support forum. There are plenty of
ways to do what you need. Start by reading about the mbstring functions.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=63732
--
Edit this bug report at https://bugs.php.net/bug.php?id=63732&edit=1
Thread (9 messages)