Unicode::LineBreak#

📦 std

Line breaking for Unicode text, and strings of grapheme clusters.

Unicode::LineBreak folds text into lines following UAX #14, the Unicode Line Breaking Algorithm: it knows where a line may end in any script, keeps a column budget the way terminals and fixed-width fonts count it (UAX #11 East Asian widths), and lets you tailor classes, widths, line formatting and what happens to a word too long for a line. Unicode::GCString is the string type it works in: a sequence of grapheme clusters - what a reader sees as one character - with their columns and line breaking classes. Text::LineFold builds on both to fold and unfold plain text.

use Unicode::LineBreak;
use Unicode::GCString;

my $lb = Unicode::LineBreak->new(ColMax => 20);
print $lb->break("The quick brown fox jumps over the lazy dog.\n");

my $s = Unicode::GCString->new("e\x{301}\x{65E5}\x{672C}");
print $s->length, " clusters, ", $s->columns, " columns\n";
# The quick brown fox
# jumps over the lazy
# dog.
# 3 clusters, 5 columns

Scripts written without spaces between words - Thai, Lao, Khmer, Myanmar - are broken between words with the dictionaries of the system’s ICU library when it is installed (ComplexBreaking), and kept whole when it is not.

This is a native implementation of the XS half of Unicode-LineBreak 2019.001 (sombok 2.4.0); its Unicode/LineBreak.pm, Unicode/GCString.pm and Text/LineFold.pm are used unchanged.

Differences from upstream#

  • Unicode version. Classes, widths and clusters come from the interpreter’s own Unicode tables (UNICODE_VERSION is 17.0.0), and the rules are Unicode 17’s UAX #14 and UAX #29, the same perl’s \b{lb} and \X follow. Upstream is frozen at Unicode 8.0.0. So emoji are wide, emoji modifier and ZWJ sequences are one cluster, Indic conjuncts join (UAX #29 GB9c), and the Unicode 9-17 classes (EB, EM, ZWJ, HH, AK, AP, AS, VF, VI) exist; their LB_* constants follow upstream’s 40.

  • ViramaAsJoiner is accepted and reported but changes nothing: what it did beyond UAX #29 is UAX #29 itself now (GB9c).

  • SA text (Thai and its neighbours) is segmented by ICU’s dictionaries instead of libthai, so SouthEastAsian::supported() names ICU and all four scripts, and Thai words may break differently.

  • LB_CJ and LB_XX are the classes their names say. Upstream’s constants swap them: its library numbers CJ 39 and XX 38 while the constants it exports give CJ 38 and XX 39.

  • A string that is not valid UTF-8 where a string or GCString is expected ($gc . "\xE9") is read with U+FFFD for each bad sequence; upstream loops forever.

  • A Prep pattern that matches the empty string keeps the character after the match; upstream drops it.

  • break breaks a whole text in one pass instead of 1000-character slices; the lines are the same.

  • A setting call ($lb->config(Key => $value)) returns undef; upstream returns a null pointer to Perl.

Modules#

Functions#

Line breaking#

lbrule#

Obsolete: the breaking action between two classes; use breakingRule.

breakingRule#

The breaking action between the end of one string and the start of another: MANDATORY, DIRECT (a break may come there), INDIRECT (only if spaces come between them) or PROHIBITED.

reset#

Forget a partially broken text, starting afresh.

strsize#

Obsolete: len plus the columns of spc and str; use Unicode::GCString::columns.

break#

Break a whole text into lines: in list context the lines as Unicode::GCStrings, in scalar context the text with line breaks.

break_partial#

Break text arriving in pieces: each call returns the lines that are complete so far; undef ends the text and returns the rest.

Configuration#

copy#

A copy of the object: its settings, tailoring and any partially broken text.

Properties#

EAWidths#

The East_Asian_Width values, in the order of their EA_* constants.

LBClasses#

The line breaking classes, in the order of their LB_* constants.

Overload glue#

as_hashref#

The object’s user hash: what $lb->{key} reaches.

as_scalarref#

A reference to the object’s name, “Unicode::LineBreak(0x…)”.

as_string#

The object’s name, “Unicode::LineBreak(0x…)”.

Library info#

UNICODE_VERSION#

The Unicode version of the character data, “17.0.0”.

SOMBOK_VERSION#

The version of the sombok library this module reimplements.