Poison Piggy - After Action Report

Introduces reading Pi Lanningham's Poison Piggy after action report verbatim, from the SundaeSwap, Acropolis, and Pragma founder.

Reads Pi's intro: on November 21st Cardano suffered a 14 hour degradation of service, presented as cold facts with no excuses.

He is not precious about the down label and asks people to form their own opinion in good faith, not from a Twitter talking head.

A serialization bug caused a unidirectional soft fork, delaying transactions up to 400 seconds but never stopping the chain.

Block times grew to 16 minutes and 3.3 percent, 479 of 14383 transactions, missed the dominant fork, exposing bridges to replay risk.

The three founding entities, Intersect, and Sunday Labs collaborated to recommend a fork to SPOs, a decentralized process with no central reset.

Begins Pi's timeline: on November 20th a transaction on the preview testnet was accepted by some nodes and rejected by others.

Breaking the testnet is white hat by design, since the testnet exists to be broken and such finds earn bug bounties.

Homer J, later the mainnet attacker, tagged Pi in the SPO channel because a Sunday swap block was involved, ultimately a red herring.

Sunday ran older software and Cardano DB Sync crashed on a serialization error, blinding the chain explorers.

Comparing chain tips confirmed a fork correlated to node versions, and Ashish from CardanoScan reopened their infrastructure to restore visibility.

Sam Leathers coded a diagnostic tool on Amaru libraries, and with Andrew Westberg they found a delegation certificate with a hash twice too big.

Recounts being told at Milos restaurant in London that the testnet forked, realizing a race to patch mainnet had begun.

You do not broadcast the vulnerability on Twitter but instead tell SPOs through responsible disclosure to move to a safe node version.

Homer J, a member of the fake Fred Discord, knew a fix existed yet reverse engineered the exploit and aimed it at Charles's RATS pool.

Argues any experienced SPO would know the attack forks the chain, so deploying it on mainnet was a deliberate choice, not innocence.

Overnight IOG, the Cardano Foundation, Intersect, exchanges, and SPOs upgraded to the patch in a war room call fest.

IOG, the Cardano Foundation, Intersect, exchanges, and SPOs chose the more restrictive chicken chain over the faster pig chain to avoid a week of off chain wallet and explorer disruption.

The K parameter makes blocks immutable after about 12 hours, requiring a manual purge script they wrote in the war room.

At 5:16 PM the chicken chain overtook the pig chain and nodes switched on their own, a full organic recovery.

Gives the root cause: a November 2024 commit accepted overly long hashes and truncated them instead of rejecting them.

Ouroboros predicted the exact recovery time hours in advance, something no other cryptocurrency can do.

A Lamport clock effect: the pig fork's time expanded as participation fell while the chicken fork's contracted, aiding recovery.

The Cardano Foundation's canary transactions measured inclusion delays spiking to 450 seconds before converging back to 20.

The Leios team's mempool experiments showed transactions reaching both chains, which prevented most double spends.

Gives chain statistics: 846 orphan blocks on the pig chain, and only 479 transactions, 3.33 percent, missed the winning chain.

Praises Pi for creating a common vocabulary and begins the taxonomy of large scale blockchain disruptions.

Defines a soft fork as one chain's accepted blocks being a superset of another's, which Nakamoto consensus can heal automatically.

Defines a hard fork as incomparable block sets that never converge without manual intervention, like Solana's historic forks.

The worst category, a sovereignty violation of forged signatures or stolen funds, with Bitcoin's 2018 near miss and a quantum attack as examples.

Recalls Bitcoin's August 2010 value overflow that briefly created 184 billion Bitcoin, fixed by a soft fork in 19 hours, longer than Cardano's 14.

Cites the Ethereum DAO hack as a consensus violation hard fork where the Ethereum Foundation strong armed miners, which drew him into Ethereum Classic.

Classifies the incident as a large chain reorg soft fork, in the same category as Polygon's 2022 and 2023 reorgs.

Lists a full consensus halt, seen on Solana, the Binance Smart Chain bridge, and the Avalanche gossip bug, all needing manual restarts.

Classifies the event as serious but not existential, with the main concern being replay risk for exchanges stuck on the wrong fork.

The biggest save was writing the node in memory safe Haskell, since C or C++ could have allowed key theft and total chain takeover.

Credits IOG's high assurance engineering for one major bug in eight years, invoking the surviving planes survivorship fallacy.

Praises Ouroboros elegance, noting the 2160 block rollback horizon naturally extends under stress, a protocol he designed with Aggelos.

The hot warm cold peer system reverted nodes to trusted Emurgo and Cardano Foundation bootstrap peers, which slowed the pig chain.

Praises node version diversity as both cause and cure, with thousands of SPOs notified across Discord, Slack, Telegram, and Twitter, and Andrew Westberg a hero.

Lists shortcomings: users had worse experiences than the canary showed, and every chain explorer over relies on Cardano DB Sync.

Says Sebastian G of DC Spark and Midnight favors codegen and fuzzing from spec, and warns premature alternative nodes risk hard forks.

Sebastian leads the Cardano blueprint to formally specify a node before building alternatives, with a certification program and canary net.

Proposes an AI upgrade sentinel for every stake pool operator to reason through upgrades before applying them, worth treasury funding.

Endorses his pub sub proposal, which would have let Intersect broadcast an emergency upgrade channel to all SPOs.

Closes that Cardano is one chain again and stronger, entering 2026 aiming to be the most reliable distributed system ever built.