Backrunner – next generation HTTP proxy for Elliptics distributed storage

Elliptics is a powerful distributed storage for medium and large data, but it is rather low-level. It doesn’t know about ACL or REST API for example, I would compare it to block level in Linux filesystem hierarchy. In particular, Elliptics only provides C/C++, Python and Golang API bindings.

For the vast majority of the users HTTP REST API is a must, thus we created Backrunner – a new swiss-knife HTTP proxy for Elliptics distributed storage. It supports ACL, automatic bucket selection based on disk and network speed, errors, amount of free space, automatic defragmentation, header extension, local static files handling and provides simple REST API for clients.

We call Backrunner an entry point to Elliptics distributed storage. It not only provides externally visible interfaces, but also takes many administrative tasks like running defragmentation, showing properly crafted monitoring stats and so on.

Backrunner’s load balancing operates in real-time, for example it gathers upload metrics (speed, latency, errors) on every request to properly tune algorithm placing data around the cluster. It also takes into account amount of free space, disk activity, internal errors, timings from other clients, network speed and many other metrics.

We will extend it to run basic recovery operations, right now Backrunner detects that replicas are out of sync, but do not run recovery because this will likely heavily affect timings, which is generally a bad idea. That’s why Elliptics is an eventually consistent system – we pay this price for the highest possible scalability levels.

Backrunner is also distributed in docker images: https://registry.hub.docker.com/u/reverbrain/backrunner/
Documentation: http://doc.reverbrain.com/backrunner:backrunner

Tutorial is coming, stay tuned!

4 thoughts on “Backrunner – next generation HTTP proxy for Elliptics distributed storage

  1. agent

    Looks very impressive. Is it opensource (can’t find it on github)?

    1. agent

      Cool! When we use this proxy do we assume that we have lots of groups in elliptics?

  2. zbr Post author

    It doesn’t matter, backrunner works with buckets – logical objects created on top of set of replicas.
    It is up to administrator to decide how many groups he prefers.

Comments are closed.