This time I ran download test, where HTTP fastcgi elliptics network proxy downloaded object from the network itself and sent it to client.

We uploaded several tens of thousands files from 4 to 15 kb each into 4-servers storage, each server hosted two elliptics node attached to dedicated SCSI disk.
Proxy server has 2 physical 2.3 Ghz CPUs (+2 HT CPUs) and 8 Gb of RAM. Storage severs and fastcgi proxy are connected over 1 GigE network. I started 4 lighttpd dispatching workers and 200 elliptics network fastcgi daemons. There was fair number of network stack tunables we touched because of problems found in redirect test, which I will describe below.

I do not know what is the client software (called Tanks here), but it is able to generate HTTP GET load with (tens of) thousands requests per second rate with 250 rps steps. Graph below shows reply rate from elliptics network HTTP proxy (legend is the same as in redirect test).

Elliptics network HTTP fastcgi proxy download test

Proxy node was maxed at about 7-8k rps - all 4 CPUs (2 physical + 2 HT) were at 100%, where most of it was eaten by dispatching lighttpd processes. Active elliptics daemons ate at most 10% with turned on small logs, cookie and authentification messages generation and data forwarding itself (proxy downloaded 4-15kb objects from the elliptics storage and sent data to the client).
Elliptics storage nodes ate at most 10-12% of CPUs.

Now about problems we found during redirect test. In this test proxy node does not fetch and sent requested objects to client, instead it generates small XML data with direct URL to the object, which client can use itself.
With the same test setup system loafed at 8k rps and 50% CPU usage (most of which took caching bind9, since system was configured to return domain name instead of IP address, which is needed for correct cookie work). At this moment something cuts the balls happens, and perfromance dropps to about 300 rps, CPU usage decreases to zero, but nothing crashes or goes out of file descriptors (there is a fair number of them configured in ulimits). To fix the problem we tuned network stack (previously we regulary got warning about timewait socket table overflow) a bit, but problem remains (maybe it is related to client software, since I thought it is 8 nodes with 1024 fds ulimit, but the way perfromance drops and the fact, that it did not dissapear after ulimits were tuned says problem is somewhere else).

According to all observations we should be able to process 15k rps in described redirect test on the 2-way (2 physical + 2 HT CPUs) machines.