2023-01-23 · timestamps by ADAtainment
UPD 01/23/2023
- A postmortem on a weekend mainnet incident where nodes with incoming connections briefly shut down and the network stalled for about two minutes.
- The likely cause is a transient, emergent bug in balance code added for optimization, hard to reproduce because it depends on a global collective state.
- The important outcome is that Cardano did exactly what it was designed to do: it self healed, with no transactions, blocks, or money lost.
- Stake pool operators and a small team assembled in the middle of the night to gather logs and hunt the race condition.
- Sentinel nodes with rolling packet captures are being set up to catch a repeat, though upcoming changes like input endorsers may simply retire the affected code.
12 entries
A GitHub issue shows mainnet nodes with incoming connections shutting down, stalling the network about two minutes, first flagged by the Smaug pool.
The network always seems to break late on a Sunday or Monday night, and looks like a transient issue unlikely to be reproducible.
A team of Sam, Jared, and Arno pulled together Sunday night with SPOs to trace where the balance optimization code was tripped.
Cardano did exactly what it was designed to do when a stall occurs: the nodes recovered and the system healed itself.
It is unsatisfying without an exact cause, but distributed systems produce emergent bugs that are not locally reproducible.
Massive systems like Google, Azure, and AWS crash constantly from hardware, software, or even cosmic rays, as in the Google Bigtable case.
It was impressive how quickly SPOs like Smaug and Andrew Westberg showed up in the middle of the night to hunt the race condition.
The point of high assurance and formal methods is that the system recovers when something bad happens and nobody notices.
No transactions were lost, no blocks were dropped, no money was lost, and the network kept progressing instead of halting.
Upcoming changes like Genesis and input endorsers will alter the operating model, and may retire the affected code before the bug is even found.
Sentinel nodes with a raw packet capture dump are being set up on a one to two week rolling basis to catch any repeat.
Thanks go to the SPO community and to Jared, Sam, and Arno for going above and beyond to work the weekend.