Our generic system contains just 4 sata disks combined into software RAID10. Putting BLOB IO backend to ext4 fs and installing two nodes, we got quite surprising skyrocketed results:
2 sata storages, each contains 4-disks software RAID10
There are 10 millions of records, total of 87 Gb. Each node contains about 44 Gb of data, and it has 24 Gb of RAM. Although we flush caches prior each test run, readahead games quickly suck blob files back into ram, which I believe explains such results: 700 rps (of completely random IO) within 200 ms, 1000 rps within 300 ms.
I wonder why 4-disks SATA setup is close to 16-disks SAS storage. Looks like raid10 requires a serious tuning for larger storages, otherwise I can not explain such major hardware difference and quite similar performance numbers.