Elliptics network: authentication bits, IO/net priorities and write performance
In a meantime elliptics got basic authentication support. It is not aimed at protecting against rogue invaders, but instead to protect against configuration error, where different clusters happen to connect to each other breaking route table and data forwarding.
Auth cookie can be set in elliptics config via ‘auth_cookie’ global parameter. By default it is empty string.
Nodes with different cookies can not join each other and will not reconnect if connection failed.
Cookie is transferred over unencrypted channel and is not supposed to add security bits, but instead to prevent misconfiguration. Storage channels are not usually cryptographically protected (at least not at this level – client should encrypt content if needed, or it can be done via server-side scripting), and all ‘real’ authentication happens at higher levels.
We also added IO priorities (for those IO schedulers which support it) – server-server IO operations (like data recovery) may get lower priority. This applies to network coloring – it is possible to assign different network priorities (as in socket(7), which may turn on TOS bits for example) to server-server and client-server traffic.
And small note on write performance.
Elliptics uses pool of IO threads to do actual low-level work. It is possible to have forward progress when some of threads are blocked in long writes for example, although usually system is kind of unresponsible at those moments. Having backlocg thread to perform your work means that there will be number of reschedulings to complete single work request.
Usually this is not a problem, since requests come in queue and IO threads pick them up as soon as processor/kernel permits.
But for single write request this may introduce unneeded latency, disappearing in batch load. Latency came up to 40 ms per single request, which is not acceptable for some cases. So we lightly changed send logic (added CORK usage) to eliminate unneeded rescheduling on header/data split reads and also dropped acknowledge sending in writes – if we wrote data successfully there is a proper reply already (information about where and how record was stored), otherwise negative error code is sent in ack message.
Those changes dropped single write request latency to sub-millisecond range, which should be fine for now – this is a median time (not including disk _sync_, but counting blob write) we perform write request when we have a queue of jobs and not just a single request.
And now I plan to implement a secondary index in elliptics via server-side scripting. For the starters, I will put all wikipedia articles and add prefix search (secondary index over written names), like returning list of urls starting with en.wikipedia.org/wiki/abc*.
Also will post Elliptics vs MongoDB benchmark on huge number of small records on single node.
Stay tuned!
Elliptics network: server-side scripting for data processing engine Elliptics: server-side scripting
Comments are currently closed.