If you wish to make an apple pie from scratch, you must first invent the universe.
So for work-related reasons far too boring to mention, I’ve been thinking about databases. You’re excited already, I can tell!
Anyway what started me off on this post was this suggestion from IndieWebCamp that you shouldn’t use databases for web application data storage, but rather use the native filesystem. Now while IWC seems to be heavily focused on blogs and other small-scale social stuff as far as I can tell, I think there is a general principle waiting to be drawn forth, but I don’t think it’s what the author(s) here intend.
The page refers to this post bigging up 2d text arrays as a means of data storage and it’s at this point that I had a little realisation. More specifically where it starts waxing lyrical about the wonders of such intuitive, user-friendly tools as grep and sed, and I started wondering if I was reading the musings of a masochist.
But yeah, my realisation. Yes, databases are a nightmare, but they needn’t be, they’re just a tool being used at the wrong level. When the people at IWC talk about human-readable data, they forget that as soon as you start talking about computing, there is no such thing. You can’t take apart your hard drive and have a quick peek at your appointments for next week. It’s all in code. The reasons we think of some codings as being human-readable is because all the infrastructure to convert that code into a usable form (all the way from magnetic charges on a drive to ASCII) is standardised and ubiquitously transparent. We don’t have to think about how it happens, it just does, and until we can do that with databases, they will always be black boxes full of hardship and corruption, and we will not gain the full benefits of their powers.
So how do we get databases to the level of transparent ubiquity that ASCII files enjoy? Well, one part (standardisation) may yet prove itself beyond the wit (or more specifically, the pride) of man, but I have a suggestion to begin with.
Consider: Filesystems are data storage…..Databases are data storage…..Can I make it anymore obvious?…..oh….well….I guess I can…
No-one (well, nearly no-one) talks about filesystems anymore as rampaging beasts out to devour your precious data anymore, like they do databases. Filesystems are reliable, because they’ve had to be – everything relies on them, including databases! If databases want to go to the next level, they need to be the new filesystem. And I mean that literally, make a kernel driver and run your OS from it! Then, and only then, can databases take over the world.
This is my chaos 
shades of WinFS, among others…
Yes, I do vaguely recall that back in the day. The failure of MS to implement is a stark reminder that this is *hard*. Although, I would think that basing it on a document database rather than a relational one would be a better starting point (seeing as filesystems as we know them seem eerily similar to document databases in particular)