Unicode::Normalize#

📦 std

Unicode::Normalize - the Unicode normalization forms NFD, NFC, NFKD and NFKC (UAX #15), and the FCD and FCC forms of UTN #5.

use Unicode::Normalize qw(NFD NFC NFKC normalize checkNFC getCombinClass getCanon);

binmode STDOUT, ':encoding(UTF-8)';
my $s = "Cafe\x{301} \x{FB01}";              # e + combining acute, "fi" ligature
printf "NFC  %s (%d chars)\n", NFC($s), length NFC($s);   # Café fi (6 chars)
printf "NFD  %d chars\n", length NFD("Caf\x{e9}");        # 5 chars
printf "NFKC %s\n", NFKC($s);                             # Café fi
print  "same: ", (normalize('C', $s) eq NFC($s) ? "yes" : "no"), "\n";
print  "checkNFC: ", (defined checkNFC($s) ? checkNFC($s) || 'NO' : 'MAYBE'), "\n";
printf "class of U+0301: %d, U+00E9 = %vX\n", getCombinClass(0x301), getCanon(0xE9);
# class of U+0301: 230, U+00E9 = 65.301

Two strings that look the same can be different sequences of characters: é is one character, U+00E9, or two, e and the combining acute U+0301. Normalizing both to the same form makes them compare equal. NFD and NFKD decompose every character as far as it goes; NFC and NFKC decompose and then compose again, giving the short form. The K forms also replace compatibility characters - ligatures, full-width forms, superscripts - by their plain equivalents, which loses formatting; NFC is the form to store and compare text in.

The data are those of the Unicode version perl itself was built with (17.0.0 for perl 5.44), so normalization agrees with perl’s own character properties.

This is a native implementation of Unicode::Normalize 1.32; its Unicode/Normalize.pm is used unchanged. use Unicode::Normalize exports NFC, NFD, NFKC and NFKD; every other function named on this page is exported on request. The .pm adds, over these functions:

  • normalize($form, $string), where $form is C, D, KC, KD (or NFC …), FCD or FCC, and check($form, $string) for the quick checks.

  • FCD($string): the string unchanged when it is already in FCD, else a string in FCD.

  • normalize_partial($form, $unprocessed) and NFD_partial, NFC_partial, NFKD_partial, NFKC_partial: normalize what can be normalized now and leave the rest in $unprocessed, for text that arrives in pieces (see splitOnLastStarter).

A code point argument is converted as an unsigned integer: pass a plain non-negative integer.

Differences from upstream#

None.

Functions#

Normalization forms#

NFD#

NFD($string): Normalization Form D, the canonical decomposition with the combining marks in canonical order. NFKD($string) is Form KD, the same with compatibility decomposition. Both exported by default.

Also available as Unicode::Normalize::NFKD.

FCC#

NFC($string): Normalization Form C, canonical decomposition followed by canonical composition - the composed form, and the usual one for storing and comparing text. NFKC($string) is Form KC, from the compatibility decomposition; FCC($string) is UTN #5’s “Fast C Contiguous” form. NFC and NFKC are exported by default.

Also available as Unicode::Normalize::NFC, Unicode::Normalize::NFKC.

Quick checks#

checkNFD#

checkNFD($string), checkNFKD($string): the quick check of UAX #15, 1 when the string is in NFD (NFKD) and "" when it is not. For these forms the answer is always definite.

Also available as Unicode::Normalize::checkNFKD.

checkNFC#

checkNFC($string), checkNFKC($string): the quick check - 1 when the string is in NFC (NFKC), "" when it is not, and undef for “maybe”: a combining mark the check cannot decide on alone. checkNFC("A\x{301}") and checkNFC("B\x{301}") both say maybe, though only the first is not in NFC; for a definite answer compare $string eq NFC($string).

Also available as Unicode::Normalize::checkNFKC.

checkFCC#

checkFCD($string): 1 when the string is in FCD, "" when not. checkFCC($string): the same for FCC, with undef for “maybe”; a string not in FCD is never in FCC.

Also available as Unicode::Normalize::checkFCD.

Building blocks#

decompose#

decompose($string [, $compat]): each character replaced by its canonical decomposition, or by its compatibility decomposition when $compat is true. The combining marks are not put in canonical order: reorder(decompose($s)) is NFD($s), and reorder(decompose($s, 1)) is NFKD($s).

reorder#

reorder($string): every run of combining marks put in canonical order. Joined NFD (or NFKD) strings are NFD again after reorder: reorder(join '', @nfd_strings).

compose#

compose($string): canonical composition, without decomposing first; compose($nfd) is the NFC of an NFD string, compose($nfkd) the NFKC of an NFKD one. composeContiguous($string) composes only characters that are next to each other, which is how FCC is made.

Also available as Unicode::Normalize::composeContiguous.

splitOnLastStarter#

splitOnLastStarter($normalized): two strings, the part before the last starter (a character of combining class 0) and the rest. The rest may still change when more text follows, so text normalized in pieces is put together as:

Character data#

getCombinClass#

getCombinClass($code_point): the canonical combining class, 0 for a starter (230 for U+0301 COMBINING ACUTE ACCENT).

isExclusion#

isExclusion($code_point): whether it is a composition exclusion, a character composition never produces.

isSingleton#

isSingleton($code_point): whether it decomposes canonically to a single other character.

isNonStDecomp#

isNonStDecomp($code_point): whether its decomposition starts with a non-starter (a character of combining class other than 0).

isComp2nd#

isComp2nd($code_point): whether it can compose with the character before it (Hangul included, exclusions not). isNFC_MAYBE and isNFKC_MAYBE are the same property under its quick-check names, NFC_Quick_Check=Maybe and NFKC_Quick_Check=Maybe.

Also available as Unicode::Normalize::isNFC_MAYBE, Unicode::Normalize::isNFKC_MAYBE.

isNFD_NO#

isNFD_NO($code_point): the derived property NFD_Quick_Check=No - whether it decomposes canonically. isNFKD_NO is NFKD_Quick_Check=No: whether it decomposes at all.

Also available as Unicode::Normalize::isNFKD_NO.

isComp_Ex#

isComp_Ex($code_point): the derived property Full_Composition_Exclusion - an exclusion, a singleton or a non-starter decomposition. isNFC_NO is the same property, as NFC_Quick_Check=No; isNFKC_NO is NFKC_Quick_Check=No, which adds every character whose compatibility decomposition differs from its canonical one (U+FB01, the “fi” ligature).

Also available as Unicode::Normalize::isNFC_NO, Unicode::Normalize::isNFKC_NO.

getComposite#

getComposite($code_point, $next_code_point): the code point the two compose to (Hangul included, exclusions excluded), or undef.

getCanon#

getCanon($code_point): the full canonical decomposition as a string, or undef for a character that does not decompose canonically. getCompat($code_point) is the same for the compatibility decomposition. Hangul syllables decompose to their jamo.

Also available as Unicode::Normalize::getCompat.