Imagine a robot out on an asteroid. It picks up a rock, processes it for its metals and volatiles, then stores the metal into a warehouse that the other robots on the asteroid are also storing in and taking from.
The system as a whole has the information on how much of each metal is in the warehouse, but it’s not surfaced anywhere so a robot can act on it, such as, hey we have enough copper, or we need more tin.
If we add a central server and each robot alerts the central server as it takes or removes some metal, we have two problems. A meteor hits the central server and we’re back to where we were. And if that doesn’t happen, contention can build up on the central server. Each robot has to communicate with the central server for each warehouse change. If the server is far away or responding slowly, the robot is stuck with information that hasn’t been uploaded yet, and the other robots don’t have that info.
With that image in mind, I started designing a distributed data replication system. Using [CRDTs|https://crdt.tech/], the robot can just share the info with any robot around. Adding a log of metal being added or removed is commutative and it can never cause conflict if any robot doesn’t have the latest numbers. It can just add or remove as it receives new logs.
Many CRDTs use a vector clock, which means in every message, there’s an entry for each robot that’s sharing information. By the time we get to thousands of robots, that becomes unwieldy; forget millions or billions of robots.
[Frontier|https://manqus.com/] is a protocol and library to sidestep the vector clock, and scale CRDTs well past that point.
Post below, or contact me on LinkedIn so we can talk.
I’m building Frontier for data replication. You have 4 sites with data you need backed up and shared across all 4 sites with almost immediate delivery when your data link is up. Frontier is designed for that. Even if 2 of the sites can’t talk to each other, Frontier handles that. It relays data as part of the protocol, and resends if the data link has gone down for a bit.
Today, I’m looking for IT managers and website operators who recognize this problem. I want to talk to you about your current data replication strategy, what you love, what you hate, where it excels, where it falls short.
If I’m going to build a solution, I want it to match what’s actually useful before I go off and build an RDBMS on top, and ORM systems that should work better than existing ones can.