Auto-FFI (Peta::FFI)#

PetaPerl includes a built-in Foreign Function Interface that allows calling C library functions directly from Perl - without writing XS, without a C compiler, without any build step. This is a PetaPerl-exclusive feature with no perl5 equivalent.

Architecture#

Any library the system can dlopen is reachable from pure Perl, at run time, with nothing compiled. There are two levels, and which one you use depends only on whether someone has already written a binding for the library you need.

Raw FFI - the Peta::FFI module itself. Open a library, call any function in it by writing a type signature, declare its C object types as Perl classes, and use the memory and callback primitives that go with that. It works for every library, including one nobody has written a binding for.
A native binding - a module presenting an ordinary Perl API that reaches the C library for you, with the signatures settled in one place and the C objects tied to the lifetime of Perl values. It is built on Peta::FFI’s binding layer and is the later, faster and more comfortable form of the same reach. Peta::FFI::Libc, Peta::FFI::UUID, Math::GMP and Cairo are among the ones that ship. Where one covers your library, use it.

Raw FFI dispatches through libffi, so any signature you can spell works with no code generation and no build step. pperl also has FFI::Platypus built in, as a layer over Peta::FFI’s engine: each of the two can express everything the other can, and code written for CPAN’s FFI::Platypus runs unchanged. New code can use Peta::FFI directly.

Raw FFI#

dlopen / call / dlclose#

use Peta::FFI qw(dlopen call dlclose);

my $lib = dlopen("libm.so.6");
my $result = call($lib, "sqrt", "(d)d", 2.0);   # sqrt(2) = 1.4142...
print "sqrt(2) = $result\n";
dlclose($lib);

call parses the signature, looks the symbol up and prepares the call every time it runs, which suits a one-off call. The function may also be given by address instead of by name: a function pointer C handed out, such as an entry of a table of callbacks, has no name to look up.

Prepared functions: function / attach#

A function you call more than a few times is better prepared once. function does the parsing, the lookup and the preparation once and returns a code reference; each call then pays only for converting its arguments and its result, roughly a quarter to a third of what call costs. attach does the same and installs the result as a named sub, in the calling package unless the name contains :::

use Peta::FFI;

my $cairo   = Peta::FFI::dlopen("libcairo.so.2");
my $move_to = Peta::FFI::function($cairo, "cairo_move_to", "(odd)v");
$move_to->($cr, 10, 20);

package My::Zlib;
my $z = Peta::FFI::dlopen("libz.so.1");
Peta::FFI::attach($z, "zlibVersion", "()p");            # My::Zlib::zlibVersion
Peta::FFI::attach($z, "crc32", "(Lpu32)L", as => "crc");
print zlibVersion(), "\n";

A prepared function keeps its library loaded for as long as the code reference or the sub lives, whatever dlclose is called on. A missing symbol or a bad signature croaks when the function is prepared, not on its first call.

Type Signature Format#

The signature string "(arg_types)return_type" lists one type per position, optionally separated by blanks:

Code	C Type	Perl Mapping
`v`	`void`	return only; `1` is returned
`i`	`int`	integer
`l`	`long`	integer
`L`	`unsigned long` / `size_t`	unsigned integer
`i8` `i16` `i32` `i64`	`int8_t` … `int64_t`	integer
`u8` `u16` `u32` `u64`	`uint8_t` … `uint64_t`	unsigned integer
`b`	`bool`	true or false
`d`	`double`	float
`f`	`float`	float
`D`	`long double`	float (rounded to a double)
`cf` `cd`	`float` / `double complex`	`[re, im]`; a number is `[n, 0]`
`p`	`const char*`	string in; a returned string is copied
`s`	`char*` the caller frees	return only; copied, then freed
`P`	`void*` (mutable buffer)	output buffer; `\$buf` is written
`o`	opaque pointer	raw address as an integer
`<Class>`	a C object’s pointer	an object of a declared handle type
`{Class}`	`struct` by value	an object of a declared record class
`*{Class}`	`struct ` / `union `	the same object, by pointer
`*T`	`T *`	`\$scalar` in and out; `undef` is NULL
`T[N]` `T[]`	`T *` to an array	`\@array` in and out

Examples:

"(d)d" - one double argument, returns double (e.g. sqrt)
"(p)L" - one string argument, returns unsigned long (e.g. strlen)
"(ppi)i" - two strings + int, returns int
"()i" - no arguments, returns int (e.g. getpid)
"(o)i" - one opaque handle, returns int (e.g. sqlite3_close)
"(u64 <My::Pool>)b" - a 64-bit unsigned integer and a handle, returns bool
"(D)D" - a long double in and out (e.g. sqrtl)
"(cd)d" - a double complex in, a double out (e.g. cabs, which gives 5 for [3, 4])

An unsigned type reads its argument as an unsigned integer, so 2**64-1 passes as itself rather than as -1, and a return above 2**63 comes back unsigned.

Pointers and arrays. *T is a pointer to one value and T[N] (a fixed count) or T[] (any length) a pointer to an array, for T a number type, o, b or p (so p[] is a char **). You pass a reference, the callee reads and writes through it, and afterwards the scalar or the array holds what C left there; undef passes NULL. As a return type, *T gives a reference to the value (undef for NULL) and an array type gives an array reference.

my $m = Peta::FFI::call($libm, "frexp", "(d*i)d", 40, \my $exp);   # 0.625, $exp = 6
Peta::FFI::call($libc, "qsort", "(i32[]LLo)v", \@v, scalar @v, 4, $cmp);

P given a reference passes that scalar’s own bytes, and after the call the scalar holds what C wrote, at the same length:

my $buf = "\0" x 64;
Peta::FFI::call($libc, "snprintf", "(PLp;i)i", \$buf, 64, "%d", 42);

Given a string or a number instead, P passes a zeroed scratch buffer of that length or that many bytes (at least 16), which nobody reads back.

A variadic function lists its named parameters, then a ;, then the types this particular call passes through its .... "(iL;o)i" is ioctl(int, unsigned long, ...) passing one pointer, and "(pi;u32)i" is open with a mode.

s is for a C function that returns a string the caller must free. Perl gets a copy and the C string is freed at once, with libc’s free() unless function or attach names the library’s own free function with free => "name". A library not written in C, a Rust one for instance, must name its own.

o is the code for every C API built on opaque structs, the sqlite3 * and PGconn * and cairo_t * kind. The address travels as an ordinary Perl integer, the runtime never dereferences it, and undef or 0 passes NULL. Never reach for p to carry such a pointer: p means “copy a string until the first NUL byte”, which is both wrong and unsafe for anything that is not a string.

Typed handles#

o leaves the pointer’s lifetime to you. handle_type instead declares a Perl class for a C object type, together with the function that destroys it, and a signature can then name the type as <Class>:

Peta::FFI::handle_type("My::Schema", $lib, "schema_free");
my $compile = Peta::FFI::function($lib, "schema_compile", "(p)<My::Schema>");

my $schema = $compile->($schema_json) or die "invalid schema";
...                                   # schema_free runs when $schema goes

As a result, the returned pointer becomes an object of the class; a NULL pointer becomes undef, so a failed constructor reads as false.
As an argument, an object of the class passes its pointer, and undef passes NULL. Anything else - another class’s handle, a plain number, a released handle - croaks before C sees it.
When the last reference to an object goes, the destructor runs with its pointer, exactly once. No DESTROY method is involved, so the class is free for your own methods and subclasses.

Peta::FFI::release($schema) runs the destructor early. The object stays in its class, and passing it to C afterwards croaks.

Records: C structs and unions#

record declares a Perl class for a C struct, member by member in the order of the C declaration:

Peta::FFI::record("My::Point", x => "d", y => "d");
Peta::FFI::record("My::Rect",  origin => "{My::Point}", size => "{My::Point}");

my $p = My::Point->new(x => 1.5, y => -2);
print $p->x, "\n";                                  # 1.5
$p->y(4);                                           # set

# struct Point point_add(struct Point, struct Point);
my $sum = Peta::FFI::call($lib, "point_add", "({My::Point}{My::Point}){My::Point}", $p, $p);
# void point_scale(struct Point *, double);
Peta::FFI::call($lib, "point_scale", "(*{My::Point}d)v", $p, 2);

A member is an integer type, f, d, b, o, p (a const char * the record points to, read-only), a fixed array of numbers or o such as u8[16], or another record by value, {Class}. The layout, each member’s offset and the size and alignment of the whole, is the C compiler’s for the platform, as libffi computes it, so you never write an offset.

Class->new(member => value, ...) makes a zeroed record and sets the members given. $r->name reads a member and $r->name($value) sets it; an array member also takes ($index), ($index, $value) and an array reference. A member that is itself a record reads as a copy, an object of its class ($rect->origin->x), and is set from one. A record is a reference to the struct’s bytes.

In a signature, {Class} passes or returns the struct by value, as "(ii){Div}" does for libc’s div. *{Class} passes a pointer to the record’s own bytes, which the callee may change in place; undef is NULL. Callbacks take and return records too: closure("({My::Point})v", sub { ... }). Peta::FFI::sizeof("{My::Point}") and alignof give a type’s size and alignment, for records and every other type.

union declares a C union the same way, with every member at offset 0: the union is as large as its largest member and as aligned as its strictest. SDL’s event type is one:

Peta::FFI::record("Keysym", scancode => "i32", sym => "i32", mod => "u16", unused => "u32");
Peta::FFI::record("KeyEvent", type => "u32", timestamp => "u32", windowID => "u32",
                  state => "u8", repeat => "u8", pad => "u8[2]", keysym => "{Keysym}");
Peta::FFI::union("Event", type => "u32", key => "{KeyEvent}", padding => "u8[56]");

my $ev = Event->new;                       # SDL_Event
while (Peta::FFI::call($sdl, "SDL_PollEvent", "(*{Event})i", $ev)) {
    print $ev->key->keysym->sym, "\n" if $ev->type == 0x300;   # SDL_KEYDOWN
}

A union is passed by pointer, *{Event}, and is read and written whole by fetch and store and as a member of a record. Passing one by value, as {Event} or inside a struct passed by value, croaks: libffi has no union type, and a struct of the same size is passed differently on some platforms, which would make a wrong call rather than an error. Real C APIs nearly always pass unions by pointer.

Library variables#

symbol($lib, $name) gives the address of a function or a variable in a library. variable binds a Perl scalar to a library’s global variable: reading $$var reads the C variable at that moment, and assigning to it writes the variable.

my $optind = Peta::FFI::variable($libc, "optind", "i");
print $$optind, "\n";        # reads the C variable
$$optind = 1;                # writes it

The type is any that store can write; a p variable is read-only. The library stays loaded while the scalar lives.

errno#

Peta::FFI::errno() returns errno as the last C call through Peta::FFI left it, read before Perl could change it. As in C, it means something only when the function reported a failure:

my $fd = Peta::FFI::call($libc, "open", "(pi;u32)i", $path, 0x40, 0644);
if ($fd < 0) {
    local $! = Peta::FFI::errno();
    die "open $path: $!";
}

Opaque pointers and memory#

Most out-parameters need no memory handling of your own: *o takes a reference to a scalar the callee writes a pointer into.

Peta::FFI::call($lib, "sqlite3_open", "(p*o)i", ":memory:", \my $db);

For memory that has to outlive a call, or that C hands you by address, there is the pointer toolbox; every address in it is a plain Perl integer, as with o. alloc and free obtain and release a zeroed block. fetch($addr, $type) and store($addr, $type, $value) read and write a value of any type at an address, records and arrays included:

my $n  = Peta::FFI::fetch($ptr, "i32");
my $xy = Peta::FFI::fetch($ptr, "d[2]");           # [1.5, -2]
my $pt = Peta::FFI::fetch($ptr, "{My::Point}");    # a My::Point copy

For raw bytes, peek reads a count of them back as a Perl byte string and peek_cstr reads a NUL-terminated C string; poke writes bytes to an address. pack_ptr and unpack_ptr convert between an address and its machine-width byte representation.

Callbacks#

C can call back into Perl. closure wraps a coderef as a real function pointer, using the same signature grammar as call, and returns its address for you to pass wherever the C API wants a function. The canonical shape is a comparator or a handler:

my $cmp = Peta::FFI::closure('(oo)i', sub {
    my ($pa, $pb) = @_;
    my $a = unpack "l", Peta::FFI::peek($pa, 4);
    my $b = unpack "l", Peta::FFI::peek($pb, 4);
    return $a <=> $b;
});

Peta::FFI::call($libc, "qsort", "(oLLo)v", $buf, $n, 4, $cmp);
Peta::FFI::closure_free($cmp);

Three rules go with it. The closure stays alive until closure_free is called with that address, so free it only once the C side has been told to stop using it; freeing an address twice is harmless, as free(NULL) is. A callback that dies does not propagate, because the error would have to unwind through the C caller’s own stack frame: $@ is set, the callback returns a zero of its declared return type, and you check $@ after the C call that triggered it. And a closure may be called only on the interpreter’s own thread, during a C call Perl made: a comparator, a progress handler, a visitor. A library that calls it from a thread of its own stops the program with a message saying so, because Perl cannot run there.

Events from a library’s own threads#

A library that produces events on threads of its own, such as a device monitor or a netlink listener, queues them instead and signals through a file descriptor that some are waiting. Peta::FFI::EventSource watches that descriptor and takes the events on Perl’s thread. The library provides a function returning the descriptor, one returning the next event as an s string (NULL when the queue is empty), and a destructor declared with handle_type that stops its thread:

use Peta::FFI;
use Peta::FFI::EventSource;

Peta::FFI::handle_type('Example::Monitor', $lib, 'example_monitor_free');
my $open = Peta::FFI::function($lib, 'example_monitor_open', '(p)<Example::Monitor>');
my $fd   = Peta::FFI::function($lib, 'example_monitor_fd',   '(<Example::Monitor>)i32');
my $next = Peta::FFI::function($lib, 'example_monitor_next', '(<Example::Monitor>)s',
                               free => 'example_string_free');

my $source = Peta::FFI::EventSource->new($open->('block'), fd => $fd, next => $next);
my $guard  = $source->watch(sub { handle_event($_) for @_ });   # AnyEvent

->watch uses AnyEvent. For EV or select there are ->fd and ->drain, and for loops that take filehandles (Mojo::IOLoop, IO::Select) there is ->fh.

Library Discovery#

use Peta::FFI qw(scan);

my $libs = scan();
# Returns hashref: { "libz.so.1" => "/usr/lib/libz.so.1", ... }

my $ours = scan("/opt/example/lib");
my $lib  = Peta::FFI::dlopen($ours->{"libexample.so"});

scan maps each library’s file name to its path: every lib*.so and lib*.so.* file in the directories searched that this program can load. Linker scripts (/usr/lib/libc.so is one) and libraries of another architecture, such as 32-bit ones on a 64-bit system, are left out. A name found in more than one directory maps to the first one’s file.

scan(@dirs) searches those directories, in that order, and no others. scan() searches the directories in PETA_FFI_PATH (colon-separated, like PATH) first, then where the dynamic linker looks for a bare name: LD_LIBRARY_PATH, the directories /etc/ld.so.conf names, and /lib, /usr/lib and their 64-bit forms.

Nothing else searches. dlopen with a bare name asks the dynamic linker, which never reads PETA_FFI_PATH; a library installed in a directory of its own is found by scan and opened by the path it returns.

Native bindings#

A native binding gives you a Perl API for one C library, with argument validation, return-value conversion, and error handling done for you. No type signatures, no addresses: you call functions and methods. Each C function’s exact type is declared in the binding and called directly, which is faster than a prepared Peta::FFI function; the C objects free themselves when the Perl values holding them go away. A module loads even when its library is not installed, and the first call that needs the library dies with an install hint. A library too old to have a function the binding calls makes that call die, naming the C function.

Peta::FFI::Libc#

Direct bindings to libc functions, grouped by category:

Process: getpid, getppid, getuid, getgid, geteuid, getegid

Strings: strlen, strerror

Environment: getenv, setenv, unsetenv

Math: abs, labs

System: sleep, usleep, gethostname, uname, time, clock

File operations: access, unlink, rmdir, mkdir, chmod, chown, plus the access-mode constants F_OK, R_OK, W_OK, X_OK

use Peta::FFI::Libc qw(getpid gethostname uname);

print "PID: ", getpid(), "\n";
print "Host: ", gethostname(), "\n";

my @info = uname();
print "OS: $info[0] $info[2] ($info[4])\n";

Peta::FFI::UUID#

Bindings to libuuid (must be installed on the system):

use Peta::FFI::UUID qw(uuid_generate uuid_generate_random);

my $uuid = uuid_generate();           # e.g. "550e8400-e29b-41d4-a716-446655440000"
my $rand = uuid_generate_random();    # random-based UUID

Functions: uuid_generate, uuid_generate_random, uuid_generate_time, uuid_parse, uuid_is_null, uuid_unparse.

If libuuid is not installed, use Peta::FFI::UUID dies with a diagnostic message.

Math::GMP#

Arbitrary-precision integer arithmetic via libgmp, under the upstream Math::GMP name, so code written for the CPAN module runs unchanged:

use Math::GMP;

my $big = Math::GMP->new("123456789012345678901234567890");
my $sum = $big + 42;
print "$sum\n";

GMP integers are heap-allocated mpz_t values stored as blessed references. Operator overloading (+, -, *, /, %, **, <=>, "") works identically to the XS module.

If libgmp is not installed, use Math::GMP dies with a diagnostic message.

Cairo#

The 2D vector graphics library, under the upstream Cairo name and with the upstream method signatures, so code written for CPAN Cairo runs unchanged with no CPAN distribution installed. It is the largest native binding that ships and the one to read if you want to see the pattern at full size: Cairo::ImageSurface, Cairo::Context, Cairo::Path, Cairo::Pattern, Cairo::Matrix, Cairo::Font and Cairo::Region each have their own reference page under Cairo.

use Cairo;

my $surface = Cairo::ImageSurface->create('argb32', 200, 200);
my $cr      = Cairo::Context->create($surface);
$cr->set_source_rgb(0, 0, 0);
$cr->arc(100, 100, 40, 0, 6.28318);
$cr->fill;
$surface->write_to_png('out.png');

If libcairo is not installed, the call site croaks with an install hint rather than the module failing to load. The 2D graphics guide walks through how this binding is put together.

What raw FFI cannot do#

Derive types from a header. Nothing reads your .h files. You read the C prototype and write the matching signature yourself, and you declare each record’s and union’s members yourself, in the order and with the types of the C declaration. A signature or a record that disagrees with the real declaration is undefined behaviour rather than an error you get told about.

When to Use FFI vs Native Modules#

Scenario	Recommendation
Common CPAN module (List::Util, etc.)	The built-in native module
A library that a native binding covers	That binding (`Math::GMP`, `Cairo`, …)
A system library that nothing wraps yet	Raw FFI
Frequent calls to the same function	`function` or `attach`
One-off experiment	`call`

call prepares the call every time it runs. A prepared function from function or attach pays only for converting its arguments and its result, roughly a quarter to a third of what call costs. A native binding calls the C function directly and is faster again; Cairo’s methods take less time per call than perl5’s XS Cairo. Native modules such as List::Util, which implement the work itself, are fastest, at the speed of a built-in operator.