eblob

Eblob is an append-only low-level IO library, which saves data in blob files.

Following features are already supported:

fast append-only updates which do not require disk seeks
compact index to populate lookup information from disk
multi-threaded index reading during starеup
O(1) data location lookup time
readahead games with data and index blobs for maximum performance
multiple blob files support (tested with single blob file on block device too)
optional sha256 on-disk checksumming
2-stage write: prepare (which reserves the space) and commit (which calculates checksum and update in-memory and on-disk indexes). One can (re)write data in between without locks
usuall 1-stage write interface
defragmentation tool: entries to be deleted are only marked as removed, eblob_check will iterate over specified blob files and actually remove those blocks
automatic defragmentation, defragmentation on request
run-time sync support – dedicated thread runs fsync in background on all files on timed base
Google’s Snappy compression support

Elliptics network uses it as one of its low-level IO backends. Numbers I posted (1, 2, 3) also highlight its advantages.

Eblob can be downloaded from github.com/ioremap/eblob
Discussion group: https://groups.google.com/forum/?fromgroups=#!forum/reverbrain