Re: BreakIterator

On Mon, 04 Jun 2012 21:09:28 +0200, Stas Malyshev <smalyshev@sugarcrm.com> wrote:

I understand that, but I have no idea how to write proper rules for word boundaries, I just want to tell it "give me word boundaries" but not by saying createWordBoundaries() but by doing createIterator($type) where $type == WORD_BOUNDARIES.

Why? This makes no sense to me. Why would createIterator(WORD_BOUNDARIES) be better than BreakIterator::createWordInstance()? Especially in a dynamic language like PHP where you can do: $type = 'word'; $bi = BreakIterator::{"create" . $type . 'instance'}(NULL);

To iterate over code points, you can build a very simple RuleBasedBreakIterator -- new RuleBasedBreakIterator('.;'). See this example here: https://gist.github.com/2843005

Is there any reason not to provide this as a service for PHP user? I understand somebody who is a specialist in ICU knows that already, but most PHP users don't know this magic.

Well, the reason I didn't add it is because ICU didn't add such an iterator. I imagine the reason for that is that there are much more efficient ways to iterate over UTF-8 that don't involve a full-blown regex based text segmentation engine. In fact, ICU provides very efficient ways (with macros and simple specialized functions) to iterate over UTF-8 text in utf8.h: http://lxr.php.net/xref/THIRD_PARTY/ICU4C/source/common/unicode/utf8.h

Right now, the ICU implementation just calls Locale::getAvailableLocales(), but its description is "Gets all the available locales that has localized text boundary data." so I suppose it could return a different set in the future.

My only concern is that no other classes have getAvailableLocales() and it doesn't seem to do anything useful now, so maybe we should omit it for now?

I have no special love for it, but your statement is innacurate in one aspect -- I've added a similar function in IntlCalendar... whose implementation is basically the same: http://lxr.php.net/xref/THIRD_PARTY/ICU4C/source/i18n/calendar.cpp#getAvailableLocales I don't mind removing both though.

Another thing I notice here: why not make: $bi = BreakIterator::createWordInstance(NULL); $bi->setText($foo); into: $bi = BreakIterator::createWordInstance(NULL, $foo);

Two reasons: * it encourages bad behavior, namely not reusing the BreakIterator objects. * that's not the ICU signature. If ICU in the future adds overloads with a string in the second argument, we'll find ourselves with odd signatures.

OK, if you have to do getPartsIterator() it's fine as long as you can easily do foreach on it, since that's what one expects from iterator. I'd also add some flag that would skip or not skip whitespace, if this is possible - so for "foo bar" sometimes you want ['foo', ' ', 'bar'] and sometimes you want just ['foo', 'bar'] - does ICU support it somehow?

The BreakIterator cannot throws away text. You have to look at the rules statuses. Example: $text = 'This is a phrase... with some punctuation.'; $bi = BreakIterator::createWordInstance(NULL); $bi->setText($text); foreach ($bi->getPartsIterator() as $v) { if ($bi->getRuleStatus() > BreakIterator::WORD_NONE_LIMIT) var_dump($v); } string(4) "This" string(2) "is" string(1) "a" string(6) "phrase" string(4) "with" string(4) "some" string(11) "punctuation"

Again, having some full description of proposed API would be nice. For example, what hashCode() does?

The ICU docs only say "Compute a hash code for this BreakIterator." If I'm not mistaken from my quick glance at the source, it just returns the length of the forward rules. -- Gustavo Lopes