There can only be one.
It's been a bit since updating my progress on what I've been lately calling the "cryptic nebula" project. When I last left off I was working on building the [mobile nebula][mobile_nebula] using nix. For the moment I gave up on that dream, as flutter and nix just *really* don't get along and I don't want to get to distracted on problems that aren't critical to the actual goal.
Instead I'd like to pursue the next critical component of the system, and that's a shared filesystem. The use-case I'm ultimately trying to achieve is:
What I'm looking for is some kind of network filesystem, of which there are *many*. This document will attempt to evaluate all relevant projects and come up with the next steps. It may be that no project fits the bill perfectly, and that I'm stuck either modifying an existing project to my needs or, if things are looking really dire, starting a new project.
The ultimate use-case here is something like a self-hosted, distributed keybase filesystem; somewhere where individuals in the cluster can back up their personal projects, share files with each other, and possibly even be used as the base layer for more complex applications on top.
The individuals involved shouldn't have to deal with configuring their distributed FS, either to read from it or add storage resources to it. Ideally the FS process can be bundled together with the nebula process and run opaquely; the user is just running their "cryptic nebula" process and everything else is handled in the background.
There are some criteria for these projects that I'm not willing to compromise on; these criteria will form a low pass filter which, hopefully, will narrow our search appreciably.
The network filesystem used by the cryptic nebula must:
The last may come across as mean, but the reason for it is that I forsee the network filesystem client running on users' personal laptops, which cannot be assumed to have resources to spare.
Each criteria in the next set lies along a spectrum. Any project may meet one of thses criteria fully, partially, or not at all. For each criteria I assign a point value according to how fully a project meets the criteria, and then sum up the points to give the project a final score. The project with the highest final score is not necessarily the winner, but this system should at least give some good candidates for final consideration.
The criteria, and their associated points values, are:
* 0: No
* 1: Kind of, and there's not much of a community.
* 2: Kind of, but there is an active community.
* 3: Yes
* 0: No docs.
* 1: Incomplete or out-of-date docs.
* 2: Very well documented.
* 0: Requires an automated system to be built to handle adding/removing hosts.
* 1: Gracefully handled.
* 0: No.
* 1: Yes.
* 0: No.
* 1: Yes.
* 0: The file can no longer be updated.
* 1: One update clobbers the other, or both go through in an undefined order.
* 2: One update is disallowed.
* 3: A copy of the file containing the "losing" update is created (ie: how dropbox does it).
* 4: Strategy can be configured on the file/directory level.
* 0: File changes must be propagated before subsequent updates are allowed (fully consistent).
* 1: Files are snapshotted at some large-ish interval (eventually consistent).
* 2: File state (ie content hash, last modifid, etc) is propagated frequently but contents are only fully propagated once the file has "settled" (eventually consistent with debounce).
* 0: Only the most basic features are implemented.
* 1: Some extra features are implemented.
* 2: Fully POSIX compliant.
* 0: A finite number.
* 1: A finite number of dedicated hosts, infinite ephemeral.
* 2: Infinite hosts.
* 0: Data loss.
* 1: Reads and writes are halted.
* 2: Reads are allowed but writes are halted.
* 3: System is partially read/write, except effected parts.
* 0: Files are limited to below 1TB in size.
* 1: Directories are limited to below 100,000 files.
* 2: No limits.
* 0: Not at all, DIY.
* 1: Encrypted at rest.
* 2: Per-user encryption.
* 0: Not at all.
* 1: Permissions are only superifically enforced.
* 2: Fully enforced user/group restrictions, complex patterns, and/or POSIX ACLs.
* 0: Frequent.
* 1: Infrequent.
* 2: Essentially none.
* 0: Very complex.
* 1: Understandable with some study.
* 2: Very simple, easy to predict.
* 0: Total black box.
* 1: Basic logging.
* 2: CLI tooling.
* 3: Exportable metrics (e.g. prometheus).
With the rubric defined, let's start actually working through our options! There are many, many different possibilities, so this may not be an exhaustive list.
The Ceph File System, or CephFS, is a POSIX-compliant file system built on
top of Ceph’s distributed object store, RADOS. CephFS endeavors to provide a
state-of-the-art, multi-use, highly available, and performant file store for
a variety of applications, including traditional use-cases like shared home
directories, HPC scratch space, and distributed workflow shared storage.
TOTAL: 22
Ceph has been recommended to me by a few people. It is clearly a very mature project, though that maturity has brought with it a lot of complexity. A lot of the complexity of Ceph seems to be rooted in its strong consistency guarantees, which I'm confident it fulfills well, but are not really needed for the use-case I'm interested in. I'd prefer a simpler, eventually consistent, system. It's also not clear to me that Ceph would even perform very well in my use-case as it seems to want an actual datacenter deployment, with beefy hardware and hosts which are generally close together.
GlusterFS is a scalable network filesystem suitable for data-intensive tasks
such as cloud storage and media streaming. GlusterFS is free and open source
software and can utilize common off-the-shelf hardware.
TOTAL: 23
GlusterFS was my initial choice when I did a brief survey of DFSs for this use-case. However, after further digging into it I think it will suffer the same ultimate problem as CephFS: too much consistency for a wide-area application like I'm envisioning. The need for syncing user/groups across machines as actual system users is also cumbersome enough to make it not a great choice.
MooseFS is a Petabyte Open Source Network Distributed File System. It is easy
to deploy and maintain, highly reliable, fault tolerant, highly performing,
easily scalable and POSIX compliant.
MooseFS spreads data over a number of commodity servers, which are visible to
the user as one resource. For standard file operations MooseFS acts like
ordinary Unix-like file system.
TOTAL: 17
Overall MooseFS seems to me like a poor-developer's Ceph. It can do exactly the same things, but with less of a community around it. The sale's pitch and feature-gating also don't ingratiate it to me. The most damning "feature" is the master metadata server, which acts as a SPOF and only sort of supports replication (but not failover, unless you get Pro).
The following projects were intended to be reviewed, but didn't make the cut for various reasons.
Going through the featuresets of all these different projects really helped me focus in on how I actually expect this system to function, and a few things stood out to me:
In the end, it may be necessary to roll our own project for this, as I don't think any of the existing distributed file systems are suitable for what's needed.
-----
Published 2021-04-06