Elliptics network is a transactional append-only distributed hash table. Each transaction is either stored on the destination node or forwarded to another server.

Client has a completion callback which is invoked when remote node acks transaction. Client may ask several nodes to provide data in parallel or issue next request if the first one failed.
Nodes itself can reconfigure and change own IDs so data will be copied around to different nodes.

But up until now there was no mechanism to work with the failed transactions except than to return error to the client. So, for example if data was placed on the nodes which are not currently available because of network failure, read request will either block or return connection reset/refuse error.

In some cases it is a good idea to resend the same transaction when network or node becomes available again. That's what I imlpemented in the elliptics network - stall transactions are checked and if its resend counter and time match configured parameters, transaction will be resent. When resend counter hits zero, transaction is destroyed and error is returned to the client.
Checkout timeout for each transaction is specified in the configuration structure - this is a time after which transaction is considered stall and its resend counter should be checked, which may result either in transaction destruction or actual resend.

I combined this code with the reconnection logic. In particular it allows all transactions to reappear immediately when remote node goes back online.

It is implemented as a separate checking/recoonnecting thread, so it can be done without signal mess and can be safely used in the library.

Effectively it is the last feature of the elliptics network library itself iirc (but my be wrong :), other extensions will likely be done as external libraries/daemons (namely PAXOS locking). I will test these changes and if things go smooth will make a new release later this week.
Next step is likely to start POHMELFS porting to elliptics network storage.

Stay tuned!