LLVS: Git for App Data
If you have heard of me at all, chances are that it is through my work on the app Agenda. But I have worked on many projects through the years, and one of the more successful is Ensembles, a framework for syncing Core Data stores via iCloud and other services.
When I first developed Ensembles, I took a deep dive into decentralized data synchronization. The plan with Ensembles was to support different cloud storage services, and pure peer-to-peer sync. With no central “truth” or programmable server, this moves the equation firmly in the direction of decentralized algorithms.
After Ensembles was on the market a few years, I started to wonder about the other half of the abstraction. What Ensembles achieves is to allow any app built on Core Data to sync via virtually any data transport mechanism. It provides an escape from lock in: if you discover Dropbox is not working so well, or Firebase is charging too much, you can simply move elsewhere with a bare minimum of code-level change.
The other half — the half that I had been pondering — is the Core Data bit. The storage API, I guess you would call it. It’s great that Ensembles can free an app from lock in on the backend, but you are still stuck using just the one storage API. And with Swift going strong, there is definitely a push to investigate other possibilities, beginning with all things Codable.
This all happened a few years ago now, and culminated in a talk I delivered at CocoaHeads in Amsterdam. I proposed a framework which would handle moving app data around, and which was abstracted from both the backend data transport, and the front end storage API. Imagine, you could setup your app to use Core Data and sync via Firebase, or just as easily use Realm and sync via iCloud.
For a while, that was that, but then I started working on Agenda, and I was called upon to once again make something sync. We didn’t want to use Core Data, so I spent many months with my co-founder Alex Griekspoor, designing and building our sync system. There were many aspects that overlapped with things I had learned from Ensembles, but other things were handled differently. For example, we really needed to think carefully about merging. With a text editor, you can’t just pick the most recently edited version of a document; you really have to merge individual edits together as best you can. This process gave me new perspective on the sync problem, and led me to where we are today.
And where is that? Today I am finally delivering on that CocoaHeads talk. At least, I am releasing an open source framework which abstracts away the front end storage API, and the backend transport API. The Low-Level Versioned Store (LLVS) is a Swift framework which can be used across devices, tracking the evolution of your decentralized data set. In effect, it is Git for your app’s data.
“That sounds a bit vague” I hear you say, so let me elaborate. LLVS works like a traditional key-value store, in that you can store data blobs with an associated unique key. What makes it different is that each time you store a set of key-value pairs, you create a new version. This version is based on the previous version, which is based on the previous version, and so forth, forming a complete history of changes.
What you store as data is entirely up to you. You could put in some plist or JSON data, and use LLVS as a global UserDefaults; or make your Swift model objects Codable, and serialize those into the store to sync. You could even setup LLVS as a sync engine for Core Data, Realm, or any other Object-Relational Modeling (ORM) framework.
The version history in LLVS is very much like Git’s. A version can be based on a single predecessor, or can merge two different predecessors into one. It forms a full Directed, Acyclic Graph (DAG), just like in Git. You can create new branches, and merge them together again as needed. And LLVS gives you full 3-way merging, which allows you to determine exactly what has changed when resolving conflicts.
The last part of the equation is distribution of data. LLVS has the concept of an Exchange, which transports data to other copies of the store, across a network, or locally. You send/retrieve data with an exchange, just like you push/pull from other repositories with Git. A store can have as many exchanges as you like, so data can pass through any number of paths to get between two devices.
At this point, there are only two types of exchange that have been implemented: one works via the file system, which is mostly for testing purposes, and the other uses CloudKit. If you only need to share data between devices in the Apple universe, the CloudKit exchange is already an enormous advantage. Transporting your data between devices just got a lot easier with LLVS, and you aren’t locked in if you decide in future you would rather use a different service, or offer a choice of services to your users.
That’s LLVS in brief. There is much more to tell, but if you are interested, you are best to start walking through the project’s README on the Gitlab page. And if there is a frontend or backend you want to support, why not take a shot at adding it? It’s often only a few hundred lines of code, and you get bonus points for donating your code back to the open source community so everyone can benefit.
