Ubuntu Manpage: dirfile-encoding — dirfile database encoding schemes

Provided by: libgetdata-doc_0.11.0-13_all

NAME

       dirfile-encoding — dirfile database encoding schemes

DESCRIPTION

The Dirfile Standards indicate that RAW fields defined in the database are accompanied by binary files
containing the field data in the specified simple data type. In certain situations, it may be
advantageous to convert the binary files in the database into a more convenient form. This is
accomplished by encoding the binary file into the alternate form. A common use-case for encoding a
binary file is to compress it to save disk space. Only data is modified by an encoding scheme. Database
metadata is never encoded.

Support for encoding schemes is optional. An implementation need not support any particular encoding
scheme, or may only support certain operations with it, but should expect to encounter unknown encoding
schemes and fail gracefully in such situations.

Additionally, how a particular encoding is implemented is not specified by the Dirfile Standards, but,
for purposes of interoperability, all dirfile implementations are encouraged to support the encoding
implementation used by the GetData dirfile reference implementation, elaborated below.

An encoding scheme is local to the particular format specification fragment in which it is indicated.
This allows a single dirfile to have binary files which are stored using multiple encodings, by having
them defined in multiple fragments.

The rest of this manual page discusses specifics of the encoding framework implemented in the GetData
library, and does not constitute part of the Dirfile Standards.

THE GETDATA ENCODING FRAMEWORK

The GetData library provides an encoding framework which abstracts binary file I/O, allowing for generic
support for a wide variety of encoding schemes. Functions which may make use of the encoding framework
are:

gd_add(3), gd_add_raw(3), gd_add_spec(3), gd_alter_encoding(3), gd_alter_endianness(3),
gd_alter_frameoffset(3), gd_alter_entry(3), gd_alter_raw(3), gd_alter_spec(3), gd_flush(3),
gd_getdata(3), gd_malter_spec(3), gd_move(3), gd_nframes(3), gd_putdata(3), gd_raw_close(3),
gd_rename(3), and gd_sync(3).

Most of the encodings supported by GetData are implemented through external libraries which handle the
actual file I/O and data translation. All such libraries are optional; a build of the library which
omits an external library will lack support for the associated encoding scheme. In this case, GetData
will still properly identify the encoding scheme, but attempts to use GetData for file I/O via the
encoding will fail with the GD_E_UNSUPPORTED error code.

GetData discovers the encoding scheme of a particular RAW field by noting the filename extension of files
associated with the field. Binary files which form an unencoded dirfile have no file extension. The
file extension used by the other encodings are noted below. Encoding discovery proceeds by searching for
files with the known list of file extensions (in an unspecified order) and stopping when the first
successful match is made. Because of this, when the a field has multiple data files with different,
supported file extensions which could legitimately be associated with it, the encoding scheme discovered
by GetData is not well defined.

In addition to raw (unencoded) data, GetData supports nine other encoding schemes: text encoding, bzip2
encoding, flac encoding, gzip encoding, lzma encoding, sie (sample-index encoding), slim encoding, zzip
encoding, and zzslim encoding, all discussed below.

The text encoding and the sample-index encoding are implemented by GetData natively and need no external
library. As a result, they are always present in the library.

Out-of-place writes
Some of the encodings listed below only support writing via out-of-place writes; that is, raw files are
written in a temporary location and only moved into place when closed. As a result, writing to these
encodings requires making a copy of the whole binary data file. A further side effect of this is that a
third-party trying to concurrently read a Dirfile which is being written to using one of these encodings
usually doesn't work.

Within GetData, reading from a field so encoded after writing to it will cause writing to the temporary
file to be finished and then the file moved into place before the read occurs, which may take some time
to do. Encodings which perform out-of-place writes are: bzip2, flac, gzip, and lzma.

BZip2 Encoding
The BZip2 Encoding reads compressed raw binary files using the Burrows-Wheeler block sorting text
compression algorithm and Huffman coding, as implemented in the bzip2 format. GetData's BZip2 Encoding
scheme is implemented through the bzip2 compression library written by Julian Seward. All operations are
supported by the BZip2 Encoding, but writing occurs out-of-place. See the Out-of-place writes section
above for details.

GetData caches an uncompressed megabyte of data at a time to speed access times. A call to gd_nframes(3)
requires decompression of the entire binary file to determine its uncompressed size, and may take some
time to complete. The file extension of the BZip2 Encoding is .bz2.

FLAC Encoding
The FLAC Encoding compresses raw binary files using the Free Lossless Audio Codec. GetData's FLAC
Encoding scheme is implemented through the libFLAC reference implementation developed by Josh Coalson and
the Xiph.Org Foundation. All operations are supported by the FLAC Encoding, but writing occurs out-of-
place. See the Out-of-place writes section above for details.

The FLAC format only permits samples up to 32-bits, but the libFLAC reference codec can only handle
samples up to 24-bits. GetData gets around this by slicing data that is wider than 16-bits into multiple
channels (2, 4, or 8, depending on width). For big-ended data, the most-significant 16-bits are in
channel 0, the second 16-bits in channel 1, &c. For little-ended data, this is reversed, with the least
significant word in channel 0.

The sample rate specified in the FLAC header is ignored and may be any valid value. FLAC files written
by GetData use a sample rate of 1 Hz. The file extension of the FLAC Encoding is .flac. The Ogg
container format is not supported.

GZip Encoding
The GZip Encoding compresses raw binary files using Lempel-Ziv coding (LZ77) as implemented in the gzip
format. GetData's GZip Encoding scheme is implemented through the zlib compression library written by
Jean-loup Gailly and Mark Adler. All operations are supported by the GZip Encoding, but writing occurs
out-of-place. See the Out-of-place writes section above for details.

To speed the operation of gd_nframes(3), the GZip Encoding takes the uncompressed size of the file the
gzip footer, which contains the file's uncompressed size in bytes, modulo 2**32. As a result, using a
field with an (uncompressed) binary file size larger than 4 GiB as the reference field will result in the
wrong number of frames being reported. The file extension of the GZip Encoding is .gz.

LZMA Encoding
The LZMA Encoding reads compressed raw binary files using the Lempel-Ziv Markov Chain Algorithm (LZMA) as
implemented in the xz container format. GetData's LZMA Encoding scheme is implemented through the lzma
library, part of the XZ Utils suite written by Lasse Collin, Ville Koskinen, and Igor Pavlov. All
operations are supported by the LZMA Encoding, but writing occurs out-of-place. See the Out-of-place
writes section above for details. Writing is supported only for the .xz container format, and not for
the obsolete .lzma format, which can still be read.

Sample-Index Encoding
The Sample-Index Encoding (SIE) compresses raw binary data by replacing runs of repeated data, similar to
run-length encoding. SIE files contain binary records consisting of a 64-bit sample number followed by a
datum (the size and format of which is determined by the RAW field's data type in the format metadata).
The sample number indicates the last sample of the field which has the specified value. The first sample
with the value is the sample immediately following the data in the previous record, or sample number
zero, for the first record. Sample numbers are relative to any /FRAMEOFFSET specified in the Dirfile
metadata. All operations are supported by the Sample-Index Encoding. The file extension of the Sample-
Index Encoding is .sie.

Slim Encoding
The Slim Encoding reads compressed raw binary files using the slimlib compression library written by
Joseph Fowler. The slimlib library was developed at Princeton University to compress dirfile-like data.
GetData's Slim Encoding framework currently lacks write capabilities; as a result, the Slim Encoding does
not support function which modify binary files. The file extension of the Slim Encoding is .slm.

Using the Slim Encoding with GetData may result in unexpected, but manageable, memory usage. See the
gd_getdata(3) manual page for details.

Text Encoding
The Text Encoding replaces the binary data files with 7-bit ASCII files containing a decimal text
encoding of the data, one sample per line. All operations are supported by the Text Encoding. The file
extension of the Text Encoding is .txt.

ZZip Encoding
The ZZip Encoding reads compressed raw binary files using the DEFLATE algorithm as implemented in the
PKWARE ZIP archive container format. GetData's ZZip Encoding scheme is implemented through the zzip
library written by Tomi Ollila and Guido Draheim. The ZZip Encoding framework currently lacks write
capabilities; as a result the ZZip Encoding does not support functions which modify binary data.

Unlike most encoding schemes, the ZZip encoding merges all binary data files defined in a given fragment
into a single ZIP archive. The name of this archive is raw.zip by default, but a different name may be
specified using the second parameter to the /ENCODING directive. For example,

/ENCODING zzip archive

indicates that the ZIP archive is called archive.zip. The file extension of the ZZip Encoding is .zip.

ZZSlim Encoding
The ZZSlim Encoding is a convolution of the Slim Encoding and the ZZip Encoding. To create ZZSlim
Encoded files, first the raw data are compressed using the slim library, and then these slim-compressed
files are archived (and compressed again) into a ZIP archive. As with the ZZip Encoding, the ZIP archive
is raw.zip by default, but a different name may be specified with the /ENCODING directive.

Notably, since the archives have the same name as ZZip Encoded data, automatic encoding detection on
ZZSlim Encoded data always fails: they are incorrectly identified as simply ZZip Encoded. As a result,
an /ENCODING directive in the format file or else a GD_ZZSLIM_ENCODED flag passed to gd_open(3) is
required to read ZZSlim encoded data. The file extension of the ZZSlim Encoding is .zip.

Using the ZZSlim Encoding with GetData may result in unexpected, but manageable, memory usage. See the
gd_getdata(3) manual page for details.

AUTHOR

       This manual page was written by D. V. Wiebe <dvw@ketiltrout.net>.

NAME

DESCRIPTION

THE GETDATA ENCODING FRAMEWORK

AUTHOR

SEE ALSO