The conversation

Lessons learnedExperience excerpt · English

If you’re a clever enough engineer, you can define a file system in terms of a database, as …

… If you’re a clever enough engineer, you can define a file system in terms of a database, as evidenced by iCloud. Personally, I have used this knowledge to use Cassandra to store blobs of video for HLS streams. This buys me a lot of Cassandra’s distributed niceties, at the cost of having to sort of reinvent some file system stuff. [1] I realize that this is very simplified; I am just speaking extremely high level.

Original discussionApple built iCloud to store billions of databases189 source comments · at import
8

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 8

Direct replies available in this import. Original-language excerpts.

KZ

If I understand correctly, bcachefs, that new hot filesystem, is pretty similar to a database - maybe someone knows more about this.

Full original
MU

Thats how Amazon made Aurora. Move all state onto the object storage layer which is also at the end of processing (you go through the lb, than frontend, than backend, than database and land on disk). Stateless is basically moving everything to the back. Im pretty sure google is doing the same thing/started with it. Also this makes it 'easily' scalable horizontal: As soon as you are able to abstract on object level, you can scale your underlying infrastructure to just handle 'objects'.

Full original
PA

> there’s not really a difference between a database and a file system. That was the promise of WinFS back in the day, which would have been really something had MS managed to bring it to fruition. I still remember the hype from back then, in my opinion totally justified, too bad that things didn't come to be. I legit think that that project could have changed the face of computing as we know it today.

Full original
ME

If you look up WinFS (which is a cancelled Windows file system originally intended to ship with Windows Longhorn), its basic principle is exactly that, be a database that happens to work as a file system. Not sure why exactly it failed, I assume that it just wasn't a suitable idea at the time given that most consumer devices (especially laptops) had very slow traditional hard drives, but in the age of NVMe storage, maybe it would be worth revisiting, assuming that Microsoft is still interested in evolving Windows in meaningful ways outside of better Ad delivery mechanisms.

Full original
MI

You don't really need to be a clever engineer, there are pre-made implementations out there for you. For example if you have an Oracle DB, then it has a feature called DBFS that does this already: https://docs.oracle.com/en/database/oracle/oracle-database/2... You can instantiate a POSIX compatible FS using database tables, and then mount them using FUSE. From there you can export it via NFS if you wish. You can also export the FS via WebDAV and thus mount it over the network using the WebDAV support built in to Windows or macOS. If you want to work with the FS transactionally, you have to do that using PL/SQL. POSIX doesn't define APIs for FS transactions, so some other approach is needed. Because it's stored in the DB you can use all the other features of the RDBMS too like clustering, replication, encryption, compression and if need be, you can maintain indexes over the file content…

Full original
SH

I disagree because querying is an important feature of most databases as usually conceived, so I think filesystems are only a subset of a database.

Full original
JW

> there’s not really a difference between a database and a file system Having written an interface to FoundationDB in preparation to moving my app over to it, I couldn't disagree more. Even "has proper indexes" is not something we'd agree on. In my case, for example, I am extremely happy with the fact that my indexes are computed in my app code, in my language, and that I am not restricted to some arbitrary concept of database-determined "indexable fields" and "indexable types". Then there are correct transactions, versionstamps (for both keys and values), streaming large amounts of data, all of that in a distributed database, it's really nothing like a filesystem.

Full original
DH

I remember back in the 80s thinking that a file system that was organized like a relational database¹ would be a really wonderful thing. Files could live in multiple places with little difficulty and any sort of metadata could be easily applied to files and queried against. ⸻ 1. I had read the original paper on database normalization over the summer and was on a database high at the time. I was young.

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.