cdb (software)

For other uses, see CDB (disambiguation).

cdb, short for "constant database", refers to both a library and data format created by Daniel J. Bernstein. cdb acts as an on-disk associative array, mapping keys to values, and allows multiple values to be stored for a single key. A constant database allows only two operations: creation and reading. Both operations are designed to be very fast and highly reliable. Since the database does not change while it is in use, multiple processes can access a single database without locking. Additionally, since all modifications create a replacement database, it can take advantage of UNIX filesystem semantics to provide a guarantee of reliability.

Record positions, key and value lengths, and hash values are 32-bit quantities and therefore must fit into 4 gigabytes. cdb is used by djbdns, fastforward, mess822, qmail and ucspi-tcp to provide highly efficient, reliable, and simple data access.

Structure

A database contains an entire data set (e.g. a single associative array) in a single computer file. It consists of three parts: a fixed-size header, data, and a set of hash tables. Lookups are designed for exact keys only, though other types of searches could be performed by scanning the entire database. Lookups are performed using the following algorithm:

• Hash the key.
• Determine at which hash table and slot this record should be located.
• Test the indicated slot in the hash table.
  • If the slot is empty, the record does not exist. Abort the search.
  • If the slot's hash matches the key's hash, seek to the record. Read and compare the key. If it matches, the data has been found, so end the search.
  • The record is not in this slot. Proceed to the next slot, wrapping around to the beginning of the hash table if necessary.

For lookups of keys with multiple values, additional values may be found by simply resuming the search at the next slot.

Format

All numbers—offsets, lengths, and hash values—are unsigned 32-bit integers, stored in little endian format. Keys and data are considered to be opaque byte strings, and have no special treatment.

The fixed-size header at the beginning of the database describes 256 hash tables by listing their position within the file and their length in slots. Data is stored as a series of records, each storing key length, data length, key, and data. There are no alignment or sorting rules. The records are followed by a set of 256 hash tables of varying lengths. Since zero is a valid length, there may be fewer than 256 hash tables physically stored in the database, but there are nonetheless considered to be 256 tables. Hash tables contain a series of slots, each of which contains a hash value and a record offset. "Empty slots" have an offset of zero.

Hashes are unsigned 32-bit integers, and start with a value of 5381. For each byte of the key, the current hash is multiplied by 33, then XOR'ed with the current byte of the key. Overflow bits are discarded. Slots and tables are trivially computed from hashes. The target table is simply the lowest eight bits of the hash (i.e. hash modulo 256), and the slot within the table is the remaining bits of the hash modulo the table length (i.e. hash divided by 256 modulo table length).

Library

The official cdb library code is public domain: the individual source files are marked as such, and are also available in the public domain djbdns package. However, the rest of the cdb package used to be license-free software, meaning it must be distributed verbatim. The unusual licensing and simplicity of the format has prompted others to re-implement the library and release it under more common terms, such as Michael Tokarev's TinyCDB library, available under the public domain. There have been a few reimplementations that remove the 4 GiB limit on key/value size, however those modifications are not compatible with the original format such as this Variable width CDB which supports the original CDB file format as well as 16-bit and 64-bit versions.

In 2009, all of cdb was put in the public domain.

Notably, the creator of cdb does not intend for cdb to be used as a shared library. This differs from virtually all similar dbm-like databases, such as Berkeley DB. Many of the reimplmentations can be used as shared libraries.

External links

• cdb official cdb website
• Constant Database (cdb) Internals detailed format description
• Reimplementation and description Contains a very detailed format description in the "readme.md".
• QDBM benchmark Archived 2012-03-05 at the Wayback Machine comparing cdb against similar packages