Let's imagine a distributed hash table storage build on top of elliptics network physically spread over multiple datacenters which are geographically separated all over the world.
Let's further suppose we want to fetch some data from our local Moscow datacenter instead of main basement somewhere in Nepal. We can assign a special transformation function like dc256_sha1 for this, which will force all transaction IDs to have ff000000 as their first 4 bytes. Moscow datacenter will have 0 ID, so if there are no other nodes with IDs more than 0xff, all requests with above transformation function will point to Moscow.
So far so good, we install this function in main load balancer all over the world, but will put it somewhere at the end of the list, since in Nepal in particular people will prefer not to go to Moscow. But what if some clients in Moscow connected to main load balancer in Nepal? If it does not know how to dynamically select transformation functions from its set according to IP or query, it will try the one listed first.
This is changed now - I added two calls to external library (not counting initialization and cleanup callbacks) which will receive address and query, start one is executed before data is processed (i.e. before GET/POST handlers start) and stop one is executed at the very end (just before data is freed). Elliptics fastCGI daemon will call external library to get region ID from query and address, and if it is more or equal than 0, it will search for 'dc$id_' string (like 'dc256_' string) and if there is such transformation fucntion, it will be moved first. Stop callback will move it to the end of the list.
So, when such system is installed all over the world datacenters, their frontends will receive requests, search for appropriate hash (transformation) function and probably (if configured) use local to client storages. Currently aforementioned library to obtain region ID from IP address is not public, so this feature is a bit useless for the non-programmers.
Everything can be turned off of course - it is just a matter of proper configuration options.
This was the last task before background fsck project. Which in turn is the last one for the elliptics network (modulo bug fixes).
Then I will switch to POHMELFS to finally feel again taste of kernel programming.
I see how we got mixed up. I was assuming load balancing was happening in the client. You have the clients connecting to a load balancing server. I was thinking of a distributed client file system not the http server setup.
Sorting the list was important for a load balancer in the client. You would want to make sure a Moscow client always tried the Moscow server first before trying to get objects out of Nepal. Without the sorted list everything in the network looks like it is the same distance away.
HTTP client does not have ability to use elliptics library and specify hash to use, instead it connects to load-balancing server, which will redirect it to local storage.
Client, which is linked against elliptics library can do that itself, this ability was always presented - it has full control over its list of transformation functions, so it can always try the one associated with Moscow datacenter first.
You know about Coral, right?
http://www.coralcdn.org/
It is open source.
DNS load balancing is well known solution for this kind of task, and effectively elliptics with server-side hash function selection is kind of similar idea, except we do not use qualified names and instead select node address accordingly.
This solution in the ellipitcs network is useful for the HTTP frontends which are actual clients in this scenario - client connects using its browser to load balancer, which in turn can redirect it to another server. And HTTP frontend (which is called proxy and fastcgi daemon here) in turn will select some of its preconfigured hash functions to specify local datacenter.
So effectively it is client who selects the closest node according to its address and configuration options.
Coral is much more than DNS load balancing. It is a giant distributed hash table like you are building.
"One of Coral's key goals is to avoid ever creating 'hot spots' of very high traffic, as these might dissuade volunteers from running the software out of a fear that spikes in server load may occur. It achieves this through an indexing abstraction called a distributed sloppy hash table (DSHT); DSHTs create self-organizing clusters of nodes that fetch information from each other to avoid communicating with more distant or heavily-loaded servers.
The sloppy hash table refers to the fact that coral is made up of concentric rings of distributed hash tables (DHTs), each ring representing a wider and wider geographic range (or rather, ping range). The DHTs are composed of nodes all within some latency of each other (for example, a ring of nodes within 20 milliseconds of each other). It avoids hot spots (the 'sloppy' part) by only continuing to query progressively larger sized rings if they are not overburdened. In other words, if the two top-most rings rings are experiencing too much traffic, a node will just ping closer ones: when a node that is overloaded is reached, upward progression stops. This minimises the occurrence of hot spots, with the disadvantage that knowledge of the system as a whole is reduced.
Requests from users are directed to a relatively close node, which then finds the file on the coral DSHT and forwards it to the user."
I meant that it uses DNS to implement node selection, something similar to what I implemented with IP address geographics checks.
In Elliptics there is no automatic node setup which will allow to split them based on latency, but such utils could easily play with hash functions on clients to select the most convenient storage node.