Recipes

The conceptual chapters cover the discipline; this page is the quick-reference appendix. Each recipe is a short, self-contained answer to a common Unicode task - the kind of thing the classic Perl Unicode cookbook reaches for a CPAN module to do, but which pperl’s builtins and regex engine handle directly.

Every example here assumes your strings are already text - decoded on the way in, as the Strings and encodings chapter describes. The operations below all speak characters, so a binary string fed to them will count bytes and surprise you.

Case-insensitive sort and comparison with fc

The fc builtin returns the foldcase of a string - the same Unicode case folding the /i pattern modifier uses. Two strings are equal under case folding exactly when fc returns the same result for both, which makes fc the right tool for case-insensitive sorting and comparison.

use v5.36;     # or: use feature 'fc';
use utf8;

# sort case-insensitively
my @sorted = sort { fc($a) cmp fc($b) } @list;

# both are true:
fc("tschüß")  eq fc("TSCHÜSS");
fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ");

Folding handles the cases that simple lc cannot: the German sharp s folds to ss, and Greek final sigma ς folds together with medial σ. Reach for fc rather than lc whenever the comparison is meant to ignore case in the full Unicode sense.

This is case folding only. It does not ignore accents, and it does not impose a locale’s alphabetic order - for that you would need a collation library such as Unicode::Collate, which is not yet usable in pperl (see the end of this page).

Normalising line breaks with \R

\R matches a single Unicode linebreak: the two-character CRLF grapheme, or any one of the vertical whitespace characters (LF, CR, form feed, vertical tab, next line, line separator, paragraph separator). It is the portable way to deal with text files that arrive from different operating systems.

my $text = "line one\r\nline two\rline three\n";
$text =~ s/\R/\n/g;          # normalise every linebreak to \n

After the substitution, $text uses a single \n everywhere regardless of how the input encoded its line endings. Use \R in place of an explicit \r?\n alternation: it covers the cases that the hand-written pattern forgets.

Matching a grapheme cluster with \X

A user-visible character - a base letter together with any combining marks - is a grapheme cluster. The . metacharacter matches one code point, which may be only part of such a cluster; \X matches one whole grapheme cluster.

use utf8;

# "é" written as e + COMBINING ACUTE ACCENT is two code points,
# but one grapheme.
my $str = "cafe\x{301}";     # c a f e + combining acute
$str =~ /^.{5}$/;            # matches - five code points
$str =~ /^\X{4}$/;           # matches - four graphemes

# find a vowel plus any combining diacritics
$str =~ / (?=[aeiou]) \X /xi;

\X is the right unit whenever “character” means “what the reader sees” rather than “code point”. The recipes below build on it.

First N graphemes, reverse, and length by grapheme

These three tasks all reduce to applying \X instead of indexing by code point. They work correctly no matter which normalization form the string is in.

Grab the first five graphemes:

my ($first_five) = $str =~ /^ ( \X{5} ) /x;

Reverse a string by grapheme. Reversing by code point would detach a combining mark from its base letter and corrupt the text; collecting graphemes first keeps each cluster intact:

$str = join "", reverse $str =~ /\X/g;

Count the length of a string in graphemes. The word brûlée is six graphemes but may be up to eight code points, so length and a grapheme count can differ:

my $count = 0;
$count++ while $str =~ /\X/g;

Fine-tuning UTF-8 warnings

Perl splits UTF-8 warnings into three subclasses, so you can silence one kind of malformed-Unicode warning without silencing the others. pperl recognises all three category names:

no warnings "nonchar";      # the forbidden non-characters
no warnings "surrogate";    # UTF-16 surrogate code points
no warnings "non_unicode";  # code points above 0x10_FFFF

Disable only the subclass you have a deliberate reason to allow - for example no warnings "non_unicode" in code that manipulates code points beyond the Unicode range on purpose - and leave the rest in force so that genuine encoding faults still surface.

Decoding @ARGV

Command-line arguments arrive as bytes. If a user passes a non-ASCII argument, you must decode it before treating it as text, exactly as you would decode any other input. The portable form maps decode over @ARGV:

use Encode qw(decode);
@ARGV = map { decode('UTF-8', $_, 1) } @ARGV;

After this line, each element of @ARGV is a text string and behaves under length, regex, and case operations in characters rather than bytes. The third argument 1 (the Encode::FB_CROAK check) makes a malformed argument a fatal error rather than a silent substitution, which is usually what you want for input you cannot re-request.

Custom property by code-point range

A \p{...} property can name a subroutine you define. When the regex engine meets an unknown property \p{In_Foo}, it calls In_Foo, which returns a list of code-point ranges (one START\tEND pair per line, in hexadecimal); the property then matches any character in those ranges.

sub In_Greek { return "0370\t03FF\n0780\t07BF\n" }

"\x{3b1}" =~ /\p{In_Greek}/;     # matches - alpha is in 0370..03FF

This is the portable way to define an ad-hoc character class once and reuse it by name across patterns, without spelling out the range in every regex.

Modules pending in pperl

A handful of recipes in the classic Perl Unicode cookbook reach for the Unicode helper modules that Perl provides as compiled extensions: Unicode::Normalize (NFC, NFD, NFKC, NFKD normalization), Unicode::Collate and Unicode::Collate::Locale (alphabetic and locale-sensitive sorting, accent-insensitive comparison), Unicode::GCString (grapheme-aware substr and print-column width), Unicode::UCD (programmatic character category and numeric-value lookup), and Unicode::LineBreak (line-breaking by Unicode rules).

These modules are not available in pperl yet: loading one today fails with dynamic loading not available in this perl. Until they are, the regex \X family above covers grapheme-aware length, reversal, and fixed-count extraction (though not print-column width), and there is no substitute here for normalization or locale collation. Named characters and user-defined properties, by contrast, work today, as the recipes above show.