eblob

Eblob is an append-only low-level IO library, which saves data in blob files.

Following features are already supported:

  • fast append-only updates which do not require disk seeks
  • compact index to populate lookup information from disk
  • multi-threaded index reading during starŠµup
  • O(1) data location lookup time
  • readahead games with data and index blobs for maximum performance
  • multiple blob files support (tested with single blob file on block device too)
  • optional sha256 on-disk checksumming
  • 2-stage write: prepare (which reserves the space) and commit (which calculates checksum and update in-memory and on-disk indexes). One can (re)write data in between without locks
  • usuall 1-stage write interface
  • defragmentation tool: entries to be deleted are only marked as removed, eblob_check will iterate over specified blob files and actually remove those blocks
  • automatic defragmentation, defragmentation on request
  • run-time sync support – dedicated thread runs fsync in background on all files on timed base
  • Google’s Snappy compression support

Elliptics network uses it as one of its low-level IO backends. Numbers I posted (1, 2, 3) also highlight its advantages.

Eblob can be downloaded from github.com/ioremap/eblob