DB_File#

📦 std

Tie a Perl hash or array to a database file: a hash table, a sorted B-tree, or an array of records kept in a plain text file.

DB_File binds a variable to a file with tie. From then on ordinary hash or array syntax reads and writes the file, and the object tie returns offers more: Berkeley DB’s own methods (get, put, del, seq, sync, fd), duplicate keys, range lookups, and array methods. The last argument of tie picks one of three kinds of database:

  • $DB_HASH - a tied hash whose keys come back in no particular order. The default.

  • $DB_BTREE - a tied hash whose keys are kept sorted, by byte value or by a sort sub of your own. It can also hold several values per key, and find the first key at or after a given one.

  • $DB_RECNO - a tied array over a text file, one element per line (or per fixed-length record). The file is an ordinary text file that other programs can read and write.

Synopsis#

use DB_File;

# A sorted hash: walk it from a given key on
my $db = tie my %h, 'DB_File', 'fruit.db', O_RDWR|O_CREAT, 0640, $DB_BTREE
    or die "fruit.db: $!";
$h{$_} = length for qw(apple banana cherry);
my ($k, $v) = ('b', '');
for (my $st = $db->seq($k, $v, R_CURSOR); $st == 0; $st = $db->seq($k, $v, R_NEXT)) {
    print "$k=$v\n";                # banana=6 cherry=6
}
undef $db;
untie %h;

# A text file as an array
tie my @lines, 'DB_File', 'notes.txt', O_RDWR|O_CREAT, 0640, $DB_RECNO
    or die "notes.txt: $!";
push @lines, 'one more line';
untie @lines;

Opening a database#

tie %h, 'DB_File', $filename, $flags, $mode, $DB_HASH;
tie %h, 'DB_File', $filename, $flags, $mode, $DB_BTREE;
tie @a, 'DB_File', $filename, $flags, $mode, $DB_RECNO;

$flags are Fcntl’s O_RDONLY, O_RDWR, O_CREAT, O_TRUNC (DB_File exports them); $mode is the permission of a file it creates. The defaults are O_CREAT|O_RDWR, 0666 and $DB_HASH, so tie %h, 'DB_File', $filename opens or creates a hash database. A relative file name is made absolute when the database is opened.

tie returns the database object, or undef with $! set when the file cannot be opened. Keep the object ($db above) to call methods on it, and undef it before untie: the file is closed only when the tied variable and every copy of the object are gone.

An undef file name opens an in-memory database: it is empty when tied and gone when untied.

$DB_HASH, $DB_BTREE and $DB_RECNO are ready-made settings objects. For other settings, make one and set its keys:

my $info = DB_File::BTREEINFO->new;
$info->{compare} = sub { lc $_[0] cmp lc $_[1] };
$info->{flags}   = R_DUP;
tie my %h, 'DB_File', 'names.db', O_RDWR|O_CREAT, 0640, $info;

  • DB_File::HASHINFO takes hash, a sub returning a number for a key, and bsize, cachesize, ffactor, nelem, lorder.

  • DB_File::BTREEINFO takes compare, flags (R_DUP), prefix, cachesize, psize, lorder, maxkeypage, minkeypage.

  • DB_File::RECNOINFO takes bval, flags (R_FIXEDLEN), reclen, bfname, cachesize, psize, lorder.

Setting any other key croaks, and so does a plain value (not a reference) for hash, compare or prefix. The hash and prefix subs, bsize, cachesize, ffactor, nelem, lorder, psize, maxkeypage and minkeypage tune Berkeley DB’s own file layout, which pperl’s files do not have: they are accepted and change nothing. The hash sub is still called, once when the database opens and once for every operation on a key, and its result unused.

Sorted keys and duplicates#

A $DB_BTREE database keeps its keys in byte order unless its settings carry a compare sub. That sub gets two keys and returns less than, equal to or greater than zero, as cmp does. The file does not remember it: every later tie of the file must give the same compare, or lookups go wrong.

With flags set to R_DUP, a key may hold several values, kept in the order they were stored. Fetching the key gives its first value; keys and each return the key once per value; deleting it deletes them all. These methods reach the values one by one:

  • $db->get_dup($key) - the key’s values, as a list; in scalar context, how many there are. $db->get_dup($key, 1) returns a hash counting each distinct value instead.

  • $db->find_dup($key, $value) - 0 when the key holds that value, with the cursor left on it, so that $db->del($key, R_CURSOR) deletes just that one; 1 otherwise.

  • $db->del_dup($key, $value) - deletes that one value of the key: 0 if it was there, 1 if not.

The file remembers that it holds duplicates: a later tie without R_DUP still has them, and asking for R_DUP of a file made without it fails the tie unless O_TRUNC empties the file.

Arrays#

A $DB_RECNO array reads its text file when tied and writes it back at sync and untie, if anything changed. Each element is one line without its newline. bval sets another delimiter ("-", or a character’s number). With flags => R_FIXEDLEN and reclen, each element is that many bytes instead, a shorter one padded with bval (a space by default) and a longer one refused.

The file is created if it does not exist, and the array can always be written, whatever $flags say. Deleting an element moves the ones after it down. Storing past the end creates the elements in between as empty: they read as undef, exists is false for them, and they are written to the file as empty lines. A negative index counts from the end; one before the start croaks Modification of non-creatable array value attempted.

The array methods push, pop, shift, unshift, length and splice work on the database object as on the tied array. With no file name the array lives in memory only, or, when the settings have a bfname, in that text file.

The cursor#

Every database has one cursor, which seq moves and the methods’ R_CURSOR flag acts at. The tie interface moves it too: keys, each, pop and the like leave it somewhere else. A method that fails to move it leaves it where it was. After a delete at the cursor, R_NEXT goes on to the element after the deleted one and R_PREV to the one before.

Files#

pperl keeps a $DB_HASH or $DB_BTREE database in an LMDB file: the file named, with LMDB’s lock file, NAME-lock, beside it. The Berkeley DB library is not used, and its files cannot be read: tying a real Berkeley DB file (or a gdbm file) croaks, naming what the file is, and the file is left as it was. A key may be at most 511 bytes, 510 if it starts with a NUL byte and 503 in an R_DUP database; a longer one croaks.

Writes reach the file at sync and when the database is untied: until then the tie holds the file’s one writer slot, and another process that writes the file waits for it (readers never wait). Set $Peta::LMDB::DBM_WRITE = 'through' before tie to commit every write as it happens, for files that several processes write; see Peta::LMDB.

Constants#

DB_File exports the flags its methods take:

  • R_FIRST, R_LAST, R_NEXT, R_PREV - where seq moves the cursor.

  • R_CURSOR - seq: to the given key, or for a $DB_BTREE the first key at or after it; put and del: at the cursor.

  • R_IAFTER, R_IBEFORE - put a new element after or before the given one (arrays, and R_DUP databases).

  • R_NOOVERWRITE - put only a new key.

  • R_SETCURSOR - put, then leave the cursor on the key.

  • R_DUP - DB_File::BTREEINFO flag: keys with several values.

  • R_FIXEDLEN - DB_File::RECNOINFO flag: fixed-length records.

  • R_RECNOSYNC - accepted by sync; it is 0.

The other names it exports (BTREEMAGIC, HASHMAGIC, R_NOKEY, RET_SUCCESS and the like) are not defined: using one croaks Your vendor has not defined DB_File macro ..., as it does with the Berkeley DB versions perl links today.

$DB_File::db_version and $DB_File::db_ver give the version of the Berkeley DB interface the module follows, and $DB_File::Error the text of the last error a method reported.

Differences from upstream#

  • The files are LMDB files, not Berkeley DB files: real Berkeley DB files are refused, and pperl’s files cannot be read by perl5’s DB_File. A file that is not a DB_File database is never overwritten. Covered by t/82-native/Peta-LMDB/100-db_file.t.

  • A $DB_HASH database’s keys come in byte order, where Berkeley DB’s come in the order of its hash buckets. Neither order is promised; code that prints a hash database without sorting prints it in a different order. Covered by t/82-native/Peta-LMDB/100-db_file.t.

  • Keys are limited to 511 bytes (503 with R_DUP); Berkeley DB has no limit. Covered by t/82-native/Peta-LMDB/100-db_file.t.

  • Writes reach the file at sync and untie (or each one, with $Peta::LMDB::DBM_WRITE = 'through'), and a writer holds the file until then. Covered by t/82-native/Peta-LMDB/100-db_file.t.

  • Two ties of one $DB_BTREE file in a process must sort it the same way: a second tie with another order (a compare sub against none) fails with EINVAL. Covered by t/82-native/Peta-LMDB/100-db_file.t.

  • With no file name but a bfname, a $DB_RECNO array keeps its records in bfname as a text file, where Berkeley DB writes a database file of its own format there. Covered by t/82-native/Peta-LMDB/100-db_file.t.

  • The tuning settings and the prefix sub have no effect, and the hash sub’s result is unused.

  • $DB_File::Error carries Berkeley DB’s message texts without their BDB0123 numbers.

Everything else - status codes, the cursor, duplicates, Recno’s records and text file - behaves as with Berkeley DB, down to its edge cases. Covered by t/81-xs-native/DB_File/010-phenotype.t and DB_File’s own test suite, t/81-xs-native/DB_File/100-upstream-db-btree.t to 130-upstream-db-threads.t.

Functions#

Berkeley DB methods#

del#

Delete a key - every value of it, in an R_DUP database - or, with R_CURSOR, the one item the database’s cursor stands on.

get#

Look up a key and put its value in a variable, returning a status instead of the value - so a missing key and a stored undef can be told apart.

put#

Store a value under a key, with flags that say where and whether.

fd#

The file descriptor of the database’s file - for a $DB_RECNO array, its text file - or -1 for an in-memory database.

sync#

Write everything stored so far to the file and to disk, so that other processes - and the file after a crash - have it.

seq#

Move the database’s cursor and read the key and value it lands on: walk the database in order, or find the first key at or after a given one.

Array methods#

unshift#

Put elements at the start of a $DB_RECNO array, in the order given.

pop#

Remove the last element of a $DB_RECNO array and return it.

shift#

Remove the first element of a $DB_RECNO array and return it; the others move down one.

push#

Add elements to the end of a $DB_RECNO array.

length#

The number of elements of a $DB_RECNO array: up to and including its last element that holds a value, so scalar(@a) too. FETCHSIZE is the same method under the name the tied array calls. It moves the database’s cursor to that last element.

Filters#

filter_fetch_key#

Install a sub that rewrites every key read from the database - by keys, each or seq - before your code sees it.

filter_store_key#

Install a sub that rewrites every key on its way into the database - for storing, fetching, deleting or looking up - so the database holds the rewritten form.

filter_fetch_value#

Install a sub that rewrites every value read from the database before your code sees it.

filter_store_value#

Install a sub that rewrites every value on its way into the database, so the database holds the rewritten form.