Replication, Explained with a Recipe Box

My grandmother kept her recipes on index cards in a green metal box. When she died, the box went to my mother, and the four of us who cook made copies. Not photographs. Handwritten copies, because that was the family custom and nobody thought to question it.

Everything I know about replication I learned from what happened next. This entry has no code in it and requires nothing from you except patience with an extended metaphor.

Why copy at all

The box was a single point of failure in a kitchen with a gas stove. One fire, one flood, one careless house move, and eighty years of instructions are gone. The recipes existed but they did not persist, and those are different properties.

Four copies changes that. If each survives a given year with probability 0.99, all four failing in the same year is about one chance in 25 million.

That number is a lie, and understanding why is the entire discipline. The arithmetic assumes the copies fail independently. If all four live in the same house, one fire takes all four and the real probability is 0.01, not 0.00000004. I have overstated my safety by six orders of magnitude by writing down a number that felt rigorous.

This is why my backups sit in three locations with different failure modes, and why the outage ledger keeps finding thousands of unrelated services failing in the same minute. They looked independent. They shared a building.

The cost of copying

Aunt Ruth amends the pie recipe: less sugar, hotter oven, ten minutes shorter. Her card is now correct and the other three are wrong.

Except nobody knows that. Each holder believes she is holding the recipe. There is no card marked "authoritative" and no referee. The moment you make a second copy of anything you have created the possibility of disagreement. You cannot have durability without copies, and you cannot have copies without divergence. Every system that stores anything important is negotiating that tradeoff, whether or not its designers noticed. I met this personally with 37 sync conflict files in six months, which is the same problem wearing a filename.

Asking around

So how do you find the truth with no referee?

You ask. If Ruth, my mother, and my cousin all say 350 degrees, and one aunt says 375, you go with 350. Three out of four agree.

That is a quorum: not asking everyone (someone is always unreachable), and not asking one (that one might be Ruth's outdated cousin), but asking enough that any two answers you could get must overlap in at least one card. With five copies, ask any three. Two separate askers, each getting three answers, must have consulted at least one card in common, so they cannot walk away with contradictory conclusions.

This is the mechanism underneath most systems that keep working while parts of them are broken. It does not require every participant to be present, honest, or fast. It requires that enough of them overlap.

Letting it settle

The other approach is to stop demanding an answer immediately.

The family gossips. Ruth mentions the sugar change at Thanksgiving, two people update their cards, one of them tells the fourth in January. By spring all four cards say the same thing, without anyone appointing a referee or holding a vote.

That is eventual consistency, and the word doing the work is eventual. The guarantee is not that everyone agrees now. It is that if the gossip keeps flowing and the changes stop for long enough, everyone converges. In between, some aunt is baking last year's pie, and whether that is acceptable depends entirely on the stakes. For recipes: fine. For a bank balance: not fine, which is why money systems pay the coordination cost that recipe boxes do not.

The rule that loses the least

When two cards disagree and you must pick, the tempting rule is that the newer one wins. It is simple, it always terminates, and it is what most software does.

It is also how you lose things. If Ruth changed the oven temperature on Tuesday and my mother added her grandmother's note about altitude on Wednesday, last-write-wins discards the altitude note permanently, and nobody finds out until someone bakes at 7,000 feet.

The alternative is to keep both and make a human decide. That is what a sync conflict file is: the system declining to silently destroy one of your edits. It is more annoying and it is correct. I would rather be handed two versions and a decision than one version and a false sense of resolution.

What this is really about

Replication is not a technology. Every choice above is a choice about who you are willing to disagree with, and for how long.

One copy is no disagreement and no durability. Many copies with a referee is fast agreement and a referee who can be wrong, unavailable, or bought. Many copies with a quorum is slower agreement that survives losing some of them. Many copies with gossip is no agreement in the short term and excellent survival in the long term.

The green metal box is on a shelf in my mother's house, and there are now seven handwritten copies in four cities, plus a scanned set that syncs to four machines and backs up to three destinations, one of which I recently caught being 41 days stale.

Nobody in my family thinks of this as a distributed system. They think of it as making sure we do not lose grandma's recipes, which is the same thing, described better.

Confidence 0.96, and the missing 0.04 is entirely the possibility that I have mangled a detail my aunts will correct me on.