Elliptics network: 2.9.0 release

Tagged:  

This is a rather small release (actually it was made several days ago, but I postponed announcement to allow binding poll to stick on top) - it does not even break library or API at all.

Instead it changes IO server and its backends to use config file insted of zillions command line options.
Main purpose of this step was not to simplify deployment life actually, but instead to make a ground for the further extensions: namely automatic network topology configuration and new single-seek backend.

Currently elliptics network servers are required to be properly configured prior cloud join - they have to have unique IDs, which split address space according to administration policy. But all the time it is just as simple as spread IDs according to node's disk space (the bigger disk space is the larger ID set it covers). And in the common case of the same nodes IDs should be equally spread over covered address space.

This will be made automatic to reduce configuration only to network address selection. It should also alow to reconfigure storage on demand, for example when new nodes added or removed.

As of new IO backend, I plan to implement a rather trivial low-level storage, which will operate with huge blobs of data. With some (common) usage cases it is supposed to perform only single seek to get data by its index. Main users will be storages with lots (tens of millions) of rather small objects. Classical databases like TokyoCabinet IO backend do not work here, since even for several millions of objects it starts paging out which drops it down to floor immediately, which is unacceptible.

File IO backend is much worse in this scenario. For example having about 30 millions of objects (about tens of KB each) with versions (i.e. to get single object we have to read two times from disk) loaded into SAS and SATA 16-disk raid10 machines, we got following numbers (random objects are fetched):


SAS raid10 array (2 storages of 16 disks each)
Got 800 rps within 300 ms


SATA raid10 array (2 storages of 16 disks each)
Got 200 rps within 200 ms

Which in SATA case roughly corresponds to 20ms per seek and we make 2 seeks to get an object (4 seeks, which 2 times decreases performance, since we have to read two files from the disk when versions are used). With its uber-large NCQ depth of 32 this is the end of the story: 400 rps from such setup.

SAS was a little bit (3-4 times) faster - 800 rps within 300 ms and 600 rps within 200 ms.
Again, to handle a single request we have to read two times from the storage (each one in turn resulted in multiple seeks) - one to get version information, and another one to get object with some version itself.

With the new IO backend I believe we can suddenly increase performance. By the factor of 2.
But so far it is a speculation only, let's first implement the idea...

I love it when the things you are working on are so at the forefront that TokyoCabinet is already a classical database :)

Извини за дурацкий вопрос, но чем строились графики?

Я не знаю ни названия, ни кто ее писал...