Virtual datacenters and geographical linking in elliptics network storage

Tagged:  

Elliptics network is a distributed hash table storage, which among zillions of other features allows to implement virtual datacenters now. It is a set of nodes combined into some logical group, where nodes may or may not actually be physically groupped together.

I added a hash prefix (one byte) to implement virtual datacenters.

This feature allows to specify preconfigured prefix in every transaction ID (the first byte). If nodes are added with IDs starting from the same number as in above hash function, then every transaction will go to that ID.

Here is a virtual datacenter example.

Let's suppose we have two transformation functions setup on client: dc1_sha1 and dc2_sha1. They will produce sha1 hash of the data with the first byte set to 1 and 2 accordingly.

Now let's suppose we add two nodes with IDs being equal to 0x0100 and 0x0180 and then two nodes with 0x0200 and 0x0280 IDs. Now there will be two transaction made with above transformation functions with its first byte set either to 1 or 2, so there will be a guaranteed copy in the first set of nodes and in the second set (sets are indexed by the first ID byte, this set is a so called virtual datacenter).

Now we can implement multiple sets with different ID's first byte for every datacenter we want to work with. If we only need to have 2 copies, but there are more than 2 datacenters, machines from them can be spread between those sets of nodes. '2' is arbitrary here of course.

This feature also allows to implement geographical linking of the requests. Let's suppose some application receives data read request from Moscow, it can check whether its set of transformation functions contains the one with the ID assigned to Moscow. If there is such a function, it can be used first to obtain object ID and to fetch it from Moscow-local servers instead of going to New York, where main datacenter lives. If Moscow storage does not contain our requested object, we will use second transformation function, which will 'point' to main storage cluster.

I will extend HTTP elliptics frontend to support external library call, which will determine object ID from received URI and client address. In particular it will be possible to write some external library to parse address and its local geobase to select transformation hash according to that data, or fallback to generic mechanism of trying every preconfigured transformation function to generate ID.

When a client node connects to the network, is it possible to give it a sorted list of hash functions? Clients in Moscow get a list with the Moscow hash on top, client in London gets a list with the London hash on top, etc... Client code can be dumber, it just tries the hashes in the order provided by the server, Severs can send client new hash lists to change their behavior too.

This can be extended if there will be a need, currently server-side hashing is used (and can be obtained by the client) for server-side data copies, so if some server knows about local to Moscow transformation function, it means all transactions written through that server were stored in Moscow datacenter also.

While this may be a good idea overall, it will not allow flexible local caching, when only some subset of written data is cached on the local nodes, and frontends are configured to try to use local transformation functions when local client connects.