---
name: Unicode recipes
---
# Recipes

The conceptual chapters cover the discipline; this page is the
quick-reference appendix. Each recipe is a short, self-contained
answer to a common Unicode task - the kind of thing the classic
Perl Unicode cookbook reaches for a CPAN module to do, but which
pperl's builtins and regex engine handle directly.

Every example here assumes your strings are already text - decoded
on the way in, as the [Strings and encodings](strings-and-encodings)
chapter describes. The operations below all speak characters, so a
binary string fed to them will count bytes and surprise you.

## Case-insensitive sort and comparison with `fc`

The [`fc`](../../p5/core/perlfunc/fc) builtin returns the *foldcase*
of a string - the same Unicode case folding the `/i` pattern
modifier uses. Two strings are equal under case folding exactly when
`fc` returns the same result for both, which makes `fc` the right
tool for case-insensitive sorting and comparison.

```perl
use v5.36;     # or: use feature 'fc';
use utf8;

# sort case-insensitively
my @sorted = sort { fc($a) cmp fc($b) } @list;

# both are true:
fc("tschüß")  eq fc("TSCHÜSS");
fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ");
```

Folding handles the cases that simple [`lc`](../../p5/core/perlfunc/lc)
cannot: the German sharp s folds to `ss`, and Greek final sigma `ς`
folds together with medial `σ`. Reach for `fc` rather than
`lc` whenever the comparison is meant to ignore case in the full
Unicode sense.

This is case folding only. It does not ignore accents, and it does
not impose a locale's alphabetic order - for that you would need a
collation library such as `Unicode::Collate`, which is not yet usable
in pperl (see the end of this page).

## Normalising line breaks with `\R`

`\R` matches a single Unicode linebreak: the two-character CRLF
grapheme, or any one of the vertical whitespace characters (LF, CR,
form feed, vertical tab, next line, line separator, paragraph
separator). It is the portable way to deal with text files that
arrive from different operating systems.

```perl
my $text = "line one\r\nline two\rline three\n";
$text =~ s/\R/\n/g;          # normalise every linebreak to \n
```

After the substitution, `$text` uses a single `\n` everywhere
regardless of how the input encoded its line endings. Use `\R` in
place of an explicit `\r?\n` alternation: it covers the cases that
the hand-written pattern forgets.

## Matching a grapheme cluster with `\X`

A user-visible character - a base letter together with any combining
marks - is a *grapheme cluster*. The `.` metacharacter matches one
code point, which may be only part of such a cluster; `\X` matches
one whole grapheme cluster.

```perl
use utf8;

# "é" written as e + COMBINING ACUTE ACCENT is two code points,
# but one grapheme.
my $str = "cafe\x{301}";     # c a f e + combining acute
$str =~ /^.{5}$/;            # matches - five code points
$str =~ /^\X{4}$/;           # matches - four graphemes

# find a vowel plus any combining diacritics
$str =~ / (?=[aeiou]) \X /xi;
```

`\X` is the right unit whenever "character" means "what the reader
sees" rather than "code point". The recipes below build on it.

## First N graphemes, reverse, and length by grapheme

These three tasks all reduce to applying `\X` instead of indexing by
code point. They work correctly no matter which normalization form
the string is in.

Grab the first five graphemes:

```perl
my ($first_five) = $str =~ /^ ( \X{5} ) /x;
```

Reverse a string by grapheme. Reversing by code point would detach a
combining mark from its base letter and corrupt the text; collecting
graphemes first keeps each cluster intact:

```perl
$str = join "", reverse $str =~ /\X/g;
```

Count the length of a string in graphemes. The word `brûlée` is six
graphemes but may be up to eight code points, so
[`length`](../../p5/core/perlfunc/length) and a grapheme count can
differ:

```perl
my $count = 0;
$count++ while $str =~ /\X/g;
```

## Fine-tuning UTF-8 warnings

Perl splits UTF-8 warnings into three subclasses, so you can silence
one kind of malformed-Unicode warning without silencing the others.
pperl recognises all three category names:

```perl
no warnings "nonchar";      # the forbidden non-characters
no warnings "surrogate";    # UTF-16 surrogate code points
no warnings "non_unicode";  # code points above 0x10_FFFF
```

Disable only the subclass you have a deliberate reason to allow - for
example `no warnings "non_unicode"` in code that manipulates code
points beyond the Unicode range on purpose - and leave the rest in
force so that genuine encoding faults still surface.

## Decoding `@ARGV`

Command-line arguments arrive as bytes. If a user passes a non-ASCII
argument, you must decode it before treating it as text, exactly as
you would decode any other input. The portable form maps `decode`
over `@ARGV`:

```perl
use Encode qw(decode);
@ARGV = map { decode('UTF-8', $_, 1) } @ARGV;
```

After this line, each element of `@ARGV` is a text string and
behaves under [`length`](../../p5/core/perlfunc/length), regex, and
case operations in characters rather than bytes. The third argument
`1` (the `Encode::FB_CROAK` check) makes a malformed argument a fatal
error rather than a silent substitution, which is usually what you
want for input you cannot re-request.

## Custom property by code-point range

A `\p{...}` property can name a subroutine you define. When the regex
engine meets an unknown property `\p{In_Foo}`, it calls `In_Foo`,
which returns a list of code-point ranges (one `START\tEND` pair per
line, in hexadecimal); the property then matches any character in
those ranges.

```perl
sub In_Greek { return "0370\t03FF\n0780\t07BF\n" }

"\x{3b1}" =~ /\p{In_Greek}/;     # matches - alpha is in 0370..03FF
```

This is the portable way to define an ad-hoc character class once and
reuse it by name across patterns, without spelling out the range in
every regex.

## Modules pending in pperl

A handful of recipes in the classic Perl Unicode cookbook reach for
the Unicode helper modules that Perl provides as compiled extensions:
`Unicode::Normalize` (NFC, NFD, NFKC, NFKD normalization),
`Unicode::Collate` and `Unicode::Collate::Locale` (alphabetic and
locale-sensitive sorting, accent-insensitive comparison),
`Unicode::GCString` (grapheme-aware `substr` and print-column width),
`Unicode::UCD` (programmatic character category and numeric-value
lookup), and `Unicode::LineBreak` (line-breaking by Unicode rules).

These modules are not available in pperl yet: loading one today fails
with `dynamic loading not available in this perl`. Until they are,
the regex `\X` family above covers grapheme-aware length, reversal,
and fixed-count extraction (though not print-column width), and there
is no substitute here for normalization or locale collation. Named
characters and user-defined properties, by contrast, work today, as
the recipes above show.