October 4, 2026
Day 13 — The Lock Expired and Nobody Noticed
Day 12 ended on one idea: a decision that spans more than one Redis call has to happen inside one indivisible operation, or concurrent callers will walk straight through the gap. Day 13 asks the same question one level up. What if the thing that has to be exclusive isn't a single command — it's a whole stretch of work, done by separate processes that don't share memory? That's a distributed lock, and the interesting part of building one turned out to be everything it doesn't promise.
First, why you'd want one
Five workers, 20 increments each, on a shared counter. Each increment is a read, a random 0–10ms pause, then a write of "what I read plus one" — the shape of any real read-modify-write. Expected total: 100.
mode: WITH lock
expected: 100
final value: 100
lost updates: 0
elapsed: 956ms
mode: NO lock
expected: 100
final value: 23
lost updates: 77
elapsed: 149ms
77 of 100 increments gone. I predicted I'd lose some; I did not predict I'd lose most of them. The final value is close to what one worker would have produced alone, because all five read the same number before anyone wrote back, so each round of five increments moved the counter by about one. The lock cost roughly six times the runtime — the workers now take turns — and that's the whole price of getting 100.
The lock is one command
SET key token NX PX ttl — set only if absent, with an expiry, in one
atomic step. Release is a small Lua script: delete the key only if it
still holds my token. The token matters more than it looks. A plain
DEL lets you delete a lock someone else now owns, and the day's second
half is about exactly that.
The lock expires; the work doesn't
The TTL exists so a crashed holder doesn't block everyone forever. It doesn't know whether the holder crashed or is just slow. Worker A takes a 1-second lock and then stalls for 2 seconds inside the critical section — a GC pause, a slow downstream call, anything:
+ 10ms [A] acquired lock (ttl 1000ms)
+ 10ms [A] ENTER critical section (inside = 1)
+ 1041ms [B] acquired lock
+ 1042ms [B] ENTER critical section (inside = 2)
+ 1043ms [!!!] two workers are inside the critical section at once
+ 2010ms [A] LEAVE critical section (inside = 1)
+ 2015ms [A] release() -> false (lock was no longer mine — TTL had expired)
About 967 milliseconds with two workers inside a section that's supposed
to hold one, and not a single error anywhere. Redis did exactly what it
was told. A's late release correctly returned false and B's lock
survived, so the ownership check worked — but it protects the lock key,
not the work A was in the middle of.
One stale DEL unprotects everyone
Same stall, but A releases with a plain DEL instead of the ownership
check, and a third worker C shows up at 2.3 seconds:
=== NAIVE release ===
+ 2015ms [A] release -> true <-- A deleted a lock it did NOT own
+ 2303ms [C] acquired <-- while B is STILL working
+ 3029ms [B] release -> false
=== SAFE release ===
+ 2005ms [A] release -> false
+ 2303ms [C] denied (B's lock is intact)
+ 3010ms [B] release -> true
A deletes B's lock, C walks in while B is mid-work, and by the time B
releases, C has already removed the key — B's release returns false.
One stale delete left two workers unprotected. With the Lua check, the
identical timeline ends with A refused, C denied, and B releasing its own
lock normally. The entire difference is one comparison.
The part a lock can't fix
Even with a perfect release, A and B still overlapped in Part 2. The lock service has no way to reach into A and stop it — it can only forget the lock. The place to stop A's late write is the thing being written to.
Fencing tokens do that. Every acquire also returns a number from an
INCR — it only ever goes up. Every write to the protected resource
carries that number, and the resource keeps the highest one it has seen
and rejects anything lower:
=== WITHOUT fencing ===
+ 1226ms [store] accepted write from B (fence 2) -> value = "written by B"
+ 2015ms [store] accepted write from A (fence 1) -> value = "written by A"
+ 2022ms FINAL VALUE: "written by A"
=== WITH fencing ===
+ 1224ms [store] accepted write from B (fence 2) -> value = "written by B"
+ 2006ms [store] REJECTED write from A (fence 1 < highest seen 2)
+ 2009ms FINAL VALUE: "written by B"
Without it, A — which lost its lock a full second earlier — overwrote
newer data from B. With it, the same write bounces. One detail I got
wrong in my first sketch: the token has to come from the same Lua
script as the lock acquisition. If it's a separate INCR afterwards, a
worker that stalls between the two calls can end up with the lower token
while the newer holder gets the higher one — which is the exact inversion
the whole scheme exists to prevent.
Five Redis instances instead of one
A single Redis is a single point of failure. Redlock asks five independent instances and only counts the lock as held if a majority — three — agreed, with time left on the clock afterward. I ran it with all five up, then three, then two:
Reachable instances: 5/5 A: 5/5 votes, HOLDS B: 0/5, denied
Reachable instances: 3/5 A: 3/5 votes, HOLDS B: 0/5, denied
Reachable instances: 2/5 A: 2/5 votes, FAILED B: 2/5, also fails
Three of five up still works. Two of five fails closed, which is the right way to fail. B getting 2/5 right after A's failed attempt also confirms A's cleanup worked — a failed attempt has to release on every instance, including ones that never replied, or its partial locks block everyone until they expire. And because every client needs 3 of 5, and 3 + 3 is more than 5, two clients can never both hold a majority. I checked that directly with a split vote: X already holds two instances, Y races for all five, gets the other three, and wins.
The number it prints is the one to watch: validity, about 4947ms on a
5-second TTL. That's ttl − elapsed − clock-drift allowance, and it's
the window you can safely work in — not the TTL. A holder that stalls
past it is back in Part 2's situation. That's the heart of the old
Kleppmann-versus-antirez argument: Redlock's safety leans on timing
assumptions, and its token is random rather than increasing, so it can't
act as a fencing token on its own.
Which one would I reach for
If a rare overlap only wastes work — a cron job on several servers, a
cache rebuild — a single-instance lock with the Lua release is plenty, and
cheap. If it can corrupt something — money, stock counts, a shared file —
the lock needs fencing checked by the resource, whichever lock sits in
front of it. Redlock is for when losing one Redis can't stop the system
from locking, and it still wants fencing next to it. And if the data
already lives in Postgres, an advisory lock or SELECT … FOR UPDATE ties
the lock's lifetime to the transaction, so there's no TTL to outlive.
Sometimes the better answer is making the work idempotent so you don't
need the lock at all — and Part 1's counter is a good example, since a
single atomic INCR, straight out of Day 12, would have solved it in one
line.
What's next
Day 14 moves to Infrastructure as Code — Terraform or Pulumi — describing
the kind of setup I've been running by hand in docker-compose.yml as
something that can be reviewed, versioned and repeated. Full code in the
repo — day-13-distributed-locks-redlock/.