Eventual consistency and data correctness
With eventual consistency there is always a window when written data can not be read because of node/network failure.
Depending on how frequently recovery process starts this may be not an issue especially if data is unique.
But it is possible that there is an old copy, which is not yet updated because of failed node or because recovery is not yet completed. In this case client will sometimes get this stall data which may be not consistent with what he expected after write.
To solve this problem eventual consistency systems introduce various kinds of timestamps attached to data. Reading them from multiple nodes can tell us with some degree of probability that given data is stall or valid as well as data consistency between nodes in question.
Elliptics network is such a system – eventual consistency is maintained by external application which starts recovery process on timed base. To get always last written data we have a set of API calls, which read metadata from multiple nodes in parallel and select node(s) with the latest written data.
In theory we can ask for data written to majority of nodes instead of latest write for example, but our practice shows that this behavior is not required if not harmed.
In my sweet dreams there is a nice looking wiki-like site which hosts all documentation and examples, but this is yet to become reality. In a meantime one can grab example sources and look at how this is implemented there.
Another example (C++ this time) is how to read/write data using columns.
Python code should be self containing if read binding implementation first, which will tell what those unnamed fields are like offset, size and various flags.
Eblob got Snappy compression support Eblob got automatic defragmentation support
Comments are currently closed.
Do you know whether perl binding for your elliptics thing exists?
Nope, we had perl bindings in older releases, but dropped it, since there were no users