Awk to Perl#

You know awk. You think in pattern { action }, you split every line into $1, $2, $3 without asking, you count things with a[key]++, and you frame the whole job with BEGIN { } and END { }. Perl was written partly as “awk, but bigger” - almost everything you do in awk has a direct Perl form, and then Perl keeps going: real data structures, full regex, modules, and a language that does not run out of room when the script grows.

The one idea to carry in from the first line: Perl is a near-superset of awk. The record loop, the fields, the associative arrays, the BEGIN/END keywords, print and printf, length/substr/split

  • they all map across, often by the same name. What changes is that fields become a real array @F you can push and pop, the associative array becomes a Perl hash you can nest, and the regex engine is PCRE rather than POSIX ERE. This guide leads with the one-to-one mappings and points out the few places the analogy leaks.

Every runnable example here was executed on pperl, a Rust reimplementation of Perl 5.42; the # ... comments show its real output. Start scripts with:

use v5.42;

That one line turns on strict, warnings, and the say built-in (a print with a trailing newline), the modern feature set used throughout.

Note

One-liner switches on this pperl build awk’s natural Perl twin is the one-liner: perl -ne, perl -ane, and perl -F, -ane. Those switches are correct perl5 5.42 and are shown throughout this guide as the idiomatic form. But the implicit-loop switch family (-n, -p, -a, -F, -i, -l) is not yet recognised by this pperl build - only -e is. So the one-liner examples below are shown without output, and each is paired with an explicit-loop full-script form (while (<$fh>) { ... }) that does run on pperl today and carries the verified output. Use the full-script form until the switches land.

How to read this#

The line/record loop#

awk runs your program once per input record, with the record in $0 and the action in { ... }. Perl’s equivalent is an explicit loop with the record in $_, the implicit topic variable. The -n switch reproduces awk’s “run this block per line”; awk’s NR (record number) is Perl’s $..

awk '{ print }' input.txt

perl -ne 'print' input.txt

The runnable, full-script form of that same loop - the one that runs on pperl today - opens the file and iterates it explicitly. $_ is the current record, exactly like awk’s $0:

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {           # each line lands in $_, like awk's $0
    chomp;
    say "line $. : $_";   # $. is awk's NR
}
# line 1 : alice 85 90
# line 2 : bob 70 60
# line 3 : carol 95 100
# line 4 : alice 50 55

(The input file held alice 85 90, bob 70 60, carol 95 100, alice 50 55.) The gotcha: a line read with <$fh> keeps its trailing newline, unlike awk’s $0, which has the record separator stripped. Call chomp to strip it (chomp is the closest thing to “awk already removed the newline for you”). The record number is $. = awk’s NR; when you read several files with <>, $. keeps counting across them, so the awk FNR (per-file counter) reset is something you arrange yourself by checking eof.

Fields and splitting#

This is awk’s heart, and Perl gives you the same picture with one twist: the fields live in a real array @F, so awk’s $1 is Perl’s $F[0] (zero-based), $0 is $_, and NF is the count scalar @F. Under the -a switch Perl autosplits each line into @F exactly as awk autosplits into $1..$NF; the runnable equivalent is one explicit split.

awk '{ print $1, NF }' input.txt          # first field, field count
awk -F, '{ print $1 }' input.txt          # comma separator (FS)

perl -ane 'print "$F[0] @{[scalar @F]}\n"' input.txt   # -a fills @F
perl -F, -ane 'print "$F[0]\n"' input.txt              # -F sets the split pattern

The full-script form does the split itself. Bare split with no arguments splits $_ on whitespace and discards leading blanks - exactly awk’s default FS:

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {
    my @F = split;                          # awk's automatic field split
    say "$F[0] has ", scalar(@F) - 1, " scores"; # $F[0]=$1, @F=NF
}
# alice has 2 scores
# bob has 2 scores
# carol has 2 scores
# alice has 2 scores

Arithmetic across fields is the everyday awk action print $1, $2+$3, and it reads almost identically:

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {
    my @F = split;
    say "$F[0]: ", $F[1] + $F[2];   # awk: print $1, $2+$3
}
# alice: 175
# bob: 130
# carol: 195
# alice: 105

awk’s $NF (the last field) is Perl’s negative index $F[-1], and a non-default FS is the pattern argument to split:

use v5.42;
open my $fh, '<', '/tmp/data.csv' or die "open: $!";  # alice,85,90 ...
while (<$fh>) {
    chomp;
    my @F = split /,/;        # awk -F,
    say $F[-1];               # awk's $NF: last field
}
# 90
# 60
# 100

To join fields back with a separator (awk’s OFS), set $, (the output field separator that print/say inserts between its list arguments):

use v5.42;
open my $fh, '<', '/tmp/data.csv' or die "open: $!";
while (<$fh>) {
    chomp;
    my @F = split /,/;
    local $, = "|";           # OFS
    say @F;                   # join the list with OFS
}
# alice|85|90
# bob|70|60
# carol|95|100

The gotcha: fields are zero-based. awk’s $1 is $F[0], $2 is $F[1], and $0 (the whole record) is $_, not an element of @F. NF is scalar @F (or $#F + 1); the last field is $F[-1], not $F[NF]. And split with one space as a string (split ' ') is the special whitespace-and-strip mode; split / / (a regex with one space) is literal and does not strip leading blanks. Plain split with no argument is the awk-default mode you want most of the time.

Regex and substitution#

awk’s sub, gsub, match, and ~ all map onto Perl’s single regex syntax: =~ binds a pattern to a string, s/// substitutes, s///g is global (awk’s gsub vs sub). Perl’s engine is PCRE, a strict superset of awk’s POSIX ERE, so every awk pattern works and Perl adds backreferences, lookaround, named captures, and the /e, /r, /x flags awk has no equivalent for.

awk '/^alice/ { sub(/alice/, "ALICE"); print }' input.txt

perl -ne 'if (/^alice/) { s/alice/ALICE/; print }' input.txt

The full-script form:

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {
    next unless /^alice/;   # awk: /^alice/ { ... }
    s/alice/ALICE/;         # awk: sub(/alice/, "ALICE") (first match)
    print;
}
# ALICE 85 90
# ALICE 50 55

Use s///g for awk’s gsub (every match), and capture groups land in $1, $2 just as awk’s match populates RSTART/RLENGTH plus the gawk array. The /e flag runs arbitrary Perl in the replacement, which awk cannot do:

use v5.42;
my $s = "items: 3 and 4";
(my $t = $s) =~ s/(\d+)/$1 * 10/ge;   # /e: replacement is Perl code
say $t;                               # items: 30 and 40

length, substr, index, and the case functions map by name - watch the indexing:

use v5.42;
my $s = "alice";
say length $s;          # 5         (awk length($s))
say substr $s, 0, 3;    # ali       (awk substr($s,1,3) - awk is 1-based!)
say uc $s;              # ALICE     (awk toupper($s))

The gotcha: substr is zero-based where awk’s substr is one-based, so awk substr($s,1,3) is Perl substr($s,0,3). And s/// edits its target in place and returns the count of substitutions, not the new string - so my $new = ($s =~ s/a/b/) puts a number in $new. Copy first ((my $t = $s) =~ s/.../.../) or use the /r flag to get a fresh string. POSIX character classes ([[:digit:]]) work, but the Perl shorthands (\d, \w, \s) are shorter and what you will see in Perl code.

Addresses and ranges#

An awk rule is pattern { action }: the action runs only on records matching the pattern. In Perl that pattern is just a condition on the statement - a postfix if testing $_. A bare /re/ matches against $_, a numeric test uses $. (awk’s NR), and awk’s range /start/,/end/ is Perl’s flip-flop operator /start/ .. /end/.

awk 'NR==2'              input.txt       # print record 2
awk '/banana/'           input.txt       # print matching records
awk '/bob/,/carol/'      input.txt       # range, inclusive

perl -ne 'print if $. == 2'            input.txt
perl -ne 'print if /banana/'           input.txt
perl -ne 'print if /bob/ .. /carol/'   input.txt

The full-script flip-flop, true from the line matching the left pattern through the line matching the right, inclusive - exactly awk’s two-pattern range:

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {
    print if /bob/ .. /carol/;   # awk: /bob/,/carol/
}
# bob 70 60
# carol 95 100

The gotcha: a bare pattern with no action in awk means “print the record”; in Perl the loop is explicit, so you write the print (or say) yourself. And in a boolean test .. is the flip-flop (range) operator, not the list-range operator (1 .. 10 builds a list). Perl tells them apart by context: in the scalar position of an if, .. is the stateful line-range selector that mirrors awk’s addr1,addr2.

I/O, pipes, in-place edit#

awk reads every file named on its command line automatically and has getline for explicit reads. Perl’s <> (the diamond operator) reads each file in @ARGV in turn - the same default - and <$fh> is the explicit getline. Writing to a command or reading from one (awk’s print | "cmd" and "cmd" | getline) is a pipe filehandle.

awk '{ print > "out.txt" }' input.txt      # redirect output
awk '{ print | "sort" }'     input.txt     # pipe to a command

use v5.42;
open my $out, '>', '/tmp/out.txt' or die "write: $!";
open my $in,  '<', '/tmp/scores.txt' or die "read: $!";
while (<$in>) {
    print {$out} $_;        # awk: print > "out.txt"
}
close $out;

For pipes, open my $fh, '|-', 'sort' writes to a command’s input and open my $fh, '-|', 'ls', '-l' reads a command’s output - awk’s | "cmd" in both directions.

awk has no built-in in-place edit; you reach for a tmp-file shuffle or GNU gawk -i inplace. Perl spells it -i on a one-liner:

perl -i -pe 's/red/crimson/' file.txt      # edit in place

The runnable equivalent today is an explicit read-modify-write - slurp the lines, transform, write back - which is exactly what -i does under the hood:

use v5.42;
my $path = '/tmp/inplace.txt';        # holds: red green blue

open my $in, '<', $path or die "read: $!";
my @lines = <$in>;                    # slurp all lines (list context)
close $in;

s/e/E/g for @lines;                   # transform each line's $_

open my $out, '>', $path or die "write: $!";
print {$out} @lines;                  # write them back
close $out;
# file now holds: rEd / grEEn / bluE

The gotcha: open returns false on failure rather than aborting, so the or die "...: $!" idiom is mandatory - $! is the system error string, awk’s silent failure made explicit. And print {$out} ... needs the braces around the filehandle when it is a variable; the bare print FH ... form (no comma) is for named handles, which is a common first-day stumble.

One-liners and switches#

The one-liner is awk’s home turf and Perl’s too. The switch family that reproduces awk: -e supplies the program, -n wraps it in a per-record loop (awk’s implicit { }), -p does the same with an auto-print at the end of each cycle, -a autosplits into @F (awk’s automatic field split), -F sets the split pattern (awk’s FS), and -l handles line endings (awk’s ORS/RS convenience). BEGIN { } and END { } are the same keywords as awk, running before and after the loop.

awk '/error/ { print }'          log.txt        # print error records
awk '{ s += $NF } END { print s }' input.txt    # sum the last field
awk -F, '{ print $1 }'           input.txt      # first CSV field

perl -ne 'print if /error/'                     log.txt
perl -ane 'END { print "$s\n" } $s += $F[-1]'   input.txt
perl -F, -ane 'print "$F[0]\n"'                 input.txt

These are shown without output because the implicit-loop switches are not recognised by this pperl build yet (see the note at the top). Each maps to a verified explicit-loop form: perl -ne 'CODE' is while (<$fh>) { CODE }, -ane adds my @F = split at the top, and BEGIN/END blocks work in a full script exactly as in awk. The runnable “sum a column with BEGIN/END” form:

use v5.42;
my $n = 0;
my $sum = 0;
BEGIN { say "start" }              # runs before the loop, like awk BEGIN
END   { say "lines=$n sum=$sum" }  # runs after, like awk END
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {
    my @F = split;
    $n++;
    $sum += $F[1];                 # awk: sum += $2
}
# start
# lines=4 sum=300

The gotcha: until the switches land on pperl, write the explicit loop (it runs today and is what the switches expand to anyway), and consult the One-Liners guide for the full switch catalogue and dozens of recipes - it is the highest-leverage thing to read coming from an awk background. Perl also has full subroutines, references, modules, and OO if your script outgrows a one-liner; see the Object-oriented Programming guide and the other coming-from guides.

Control flow#

awk’s control keywords map onto Perl’s almost name-for-name. awk’s next (skip to the next record) is Perl’s next; break/continue in awk loops are Perl’s last/next; awk’s for (k in arr) becomes a Perl for over keys. The braces and ; are the same; Perl adds the postfix forms that read like awk patterns.

awk '{ if ($2 < 70) next; print }' input.txt

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
while (<$fh>) {
    my @F = split;
    next if $F[1] < 70;        # awk: if ($2 < 70) next
    print;
}
# alice 85 90
# carol 95 100

The gotcha: Perl’s comparison operators come in two families and you pick by the operands’ type, where awk has only </== for both. Numbers use ==, <, >, <=>; strings use eq, lt, gt, cmp. awk decides numeric-vs-string comparison by the values at runtime; Perl makes you choose the operator. Using == on strings (or eq on numbers) is a frequent, quiet bug - "abc" == "xyz" is true in Perl (both numify to 0), which is almost never what you meant.

Variables and data#

This is the happiest mapping in the guide: awk’s associative array is a Perl hash, and the daily idioms are near-identical. awk’s count[$1]++ is Perl’s $count{$F[0]}++; for (k in count) is for my $k (keys %count); if (k in count) is exists $count{$k}. The one shift is the sigil - the whole hash is %count, but one value is $count{$key}.

awk '{ total[$1] += $2 + $3 }
     END { for (k in total) print k, total[k] }' input.txt

use v5.42;
open my $fh, '<', '/tmp/scores.txt' or die "open: $!";
my %total;
while (<$fh>) {
    my @F = split;
    $total{$F[0]} += $F[1] + $F[2];   # awk: total[$1] += $2 + $3
}
for my $name (sort keys %total) {     # awk END loop over the array
    say "$name $total{$name}";
}
# alice 280
# bob 130
# carol 195

The two-file NR==FNR idiom - load keys from the first file, then test membership in the second - is one explicit loop per file in Perl, which is clearer than the awk one-liner it replaces:

use v5.42;
my %seen;
open my $f1, '<', '/tmp/data.csv' or die "open: $!";
while (<$f1>) {
    my ($k) = split /,/;          # first field only
    $seen{$k} = 1;
}
close $f1;
open my $f2, '<', '/tmp/scores.txt' or die "open: $!";
while (<$f2>) {
    my ($k) = split;
    print if $seen{$k};           # awk: ($1 in seen)
}
# alice 85 90
# bob 70 60
# carol 95 100
# alice 50 55

The gotcha: a hash has no inherent order (awk’s for (k in arr) is also unordered, so this will not surprise you), so iterate sort keys %h when you want a stable order. Membership is exists $h{$k}, not awk’s k in arr - and merely reading $h{$k} to test it will autovivify the key into existence in some contexts, where exists never does. Numbers and strings auto-coerce in Perl as in awk, but remember the == vs eq split from Control flow.

Gotchas coming from awk#

The traps, collected for a final scan before you write:

  • Fields are zero-based. awk’s $1 is $F[0], $2 is $F[1], and the whole record $0 is $_. NF is scalar @F; the last field is $F[-1], not $F[NF].

  • <$fh> keeps the trailing newline, unlike awk’s $0. chomp to strip it.

  • substr is zero-based, where awk’s is one-based: awk substr($s,1,3) is Perl substr($s,0,3).

  • s/// returns a count, not the new string. my $x = $s =~ s/a/b/ puts the substitution count in $x. Copy first, or use the /r flag to get a new string.

  • Two comparison families. Numbers use ==/</<=>; strings use eq/lt/cmp. awk’s single ==/< is split in two, chosen by operator, not by value. "abc" == "xyz" is true (both numify to 0).

  • Bare split is the awk-default mode. split (no args) splits $_ on whitespace and strips leading blanks. split / / (a regex with one space) is literal and does not strip - use plain split for awk’s default FS.

  • Membership is exists $h{$k}, not k in arr. Testing a key by reading $h{$k} can autovivify it; exists never does.

  • open does not abort on failure. Use open my $fh, '<', $path or die "...: $!". $! is the system error string.

  • The implicit-loop switches (-n/-p/-a/-F/-i/-l) are not yet recognised by this pperl build. Only -e works today. Write the explicit while (<$fh>) { my @F = split; ... } loop, which is exactly what the switches expand to, until they land.