I’ve just announced pohmelfs in linux-kernel

December 23, 2011 zbr Uncategorized

Here is the link: https://lkml.org/lkml/2011/12/23/161

As mentioned there, there are features yet to complete:

• quorum read. pohmelfs supports quorum write only so far.
• http compatibility mode – we do want to upload data via pohmelfs and read it through http applications. And vice versa actually too.
• column read-write or more generally file-as-directory feature.
• even more testing – abusing dcache is fun, but likely there are hiddens stones
• replace drivers/staging/pohmelfs with this code.

It is quite a lot of work, but not that much of work. Maybe a week or so to actually implement all that stuff and then to start real testing. Right now we strike into system limitations quite fast – like doing multi-terabyte rsync between local raid and pohmelfs mounted storage and hitting NIC limitations all the time – it is 1 Gbps after all (and I want to play with 10 GigE but still without luck). Pohmelfs uses local VFS page cache so it can easily saturate bandwidth during writeback.
Since we also have to write multiple copies in parallel (we write 3, but only 2 groups are turned on to get quorum), write is limited by network 40-50 MB/s.

Also elliptics sometimes scares me – we sync data into eblob once per 300 seconds, and it takes about 3-10 seconds to actually write it to disks on every system. At that time machine almost freezes – 100% disks utilization. And rsync sometimes looses its mind – it stops and timeouts, although sync is already finished.

Another problem is related to eblob. We store 10+ gigabyte log files in test case, and since eblob is append-only, it can fill its quota for given key and subsequent write will have to copy previous record into new chunk. Copying 10 Gb takes some time too, and it also takes space – old records are just marked as deleted until offline defragmentation cleans it up.

And my favourite – dcache abuse. Sometimes I get kernel BUG at /home/apw/COD/linux/fs/dcache.c:873 which means at umount time there are dentries with positive reference counters. I do not yet know whether it is pohmelfs or not, but more likely that it is the reason :)
I will fix that too of course.

But overall pohmelfs is close to finish.

Next task is to implement indexing and embedded search in elliptics.
It is quite good idea to have out-of-the box full-text search engine inside storage system.

Stay tuned, new things are close!

Elliptics, POHMELFS

POHMELFS and elliptics server-side Pohmelfs got quorum read support