We have following config for Elliptics leveldb backend:
sync = 0 root = /opt/elliptics/leveldb log = /var/log/elliptics/leveldb.log cache_size = 64424509440 write_buffer_size = 1073741824 block_size = 10485760 max_open_files = 10000 block_restart_interval = 16
And with 1kb of pretty compressable data (ascii strings) chunks pushed into single server with 128 Gb of RAM and 4 SATA disks combined into RAID10 ends up with poor 6-7 krps.
If request rate is about 20 krps median reply time is about 7 seconds (!)
dstat shows that it is not disk (well, with 20 krps it is disk), but before that it is neither CPU nor disk – leveldb just doesn’t allow more than 5-7 krps with 1 Kb data chunks from parallel threads (we have 8-64 IO threads depending on config). When snappy compression is enabled things get worse.
Is it ever possible to push 20 MB/s into LevelDB with small-to-medium (1Kb) chunks?