UPD 01/23/2023

This whiteboard video is about network operations and the balance between prevention and recovery in distributed systems.

Sunday's blip was not a crash, and a stake pool operator's tweet nicely visualized the difference between the two.

The goal is a system that works all the time, but you introduce demons that erode your prevention mechanisms.

The demons are the desire for speed and performance, open access and accessibility, permissionless operation, and Byzantine actors.

Maximizing prevention costs throughput, while making a network permissioned optimizes throughput but centralizes it.

Any good distributed system balances prevention with recovery using checkpoints, rollbacks, and restarts.

Diffusion pipelining, introduced in Vasil in 2022, made Cardano better, faster, and cheaper.

Before pipelining, a relay would fully validate a bad transaction from a user and die before propagating it.

Pipelining removes some safeguards so transactions propagate faster, so a crash can spread from one relay to its n connections.

The trade off is more speed and throughput in exchange for less Byzantine actor prevention.

Ouroboros Genesis removes the need for checkpoints, recovering from the genesis block to today, and pairs with peer to peer for recovery.

The scuttlebutt Byzantine resilient gossip protocols paper bridges the gap between idealized gossip networks in security proofs and real world attacks.

Occasionally a chain of bad transactions creates an emergent crash state that ripples across the network before recovering.

Input endorsers and the Ouroboros Leios paper are scheduled for the first half of 2023, introducing three types of blocks.

Batching transactions asynchronously into their own blocks brings enormous throughput and accessibility gains.

The new layer adds Byzantine attacks, so a clique may collapse while the main chain keeps operating fine.

You trade the perfect immune machine for occasional fault states in exchange for fast, low cost transactions.

Formal methods, modeling, and pure functions are powerful for understanding a system's behavior both locally and emergently.

Expressiveness is another demon, which is why Bitcoin does not have smart contracts.

The extended UTXO model and Plutus were designed to prevent transactions from breaking the system, which works largely as intended.

Prevention reduced expressiveness, so some developers miss Ethereum features that were left out to avoid hazards to the system.

Ethereum's sharding is not well formally modeled and leans on a bonded slash system, where a stall could slash ether at no fault of operators.

Cardano is more open with no bond and slash, which took harder protocol design so users can never lose all their money to a bad state.

Recovering in under two minutes shows a good balance between prevention and recovery given the demons the system faces.

Like a body with trillions of cells that always has some cancerous or viral cells, a complex system will always have some bad state to purge.

Cardano's balance does not require economic punitive measures to compensate for poor protocol design, unlike other systems.

The scaling and sharding plans rely on the extended UTXO model and Mithril, technologies already understood at scale.

Better network monitoring like TCP dumps helps, and while things collapsed often in 2017, Cardano has run five years with few issues.

The stall was handled entirely by the community, with no IOHK reset and no eleventh hour Discord call, since the IO team was asleep on Sunday.

The failure and recovery model needs careful thought as preventive measures are exhausted and the system scales up performance and validators.