Compact block relay

Two peers who both watch the mempool already share nearly every transaction in the next block, so sending them a second time is waste. Matt Corallo specified the fix as BIP152 and Bitcoin Core enabled it in version 0.13.0, released on August 23, 2016. For anyone running a node at home it is the difference between a tolerable bandwidth bill and an intolerable one.

How it works

Compact block relay adds four messages to the protocol, and the interesting one is cmpctblock, which describes a block without containing it.

A cmpctblock carries the 80 byte header, an 8 byte nonce chosen by the sender, and then one six byte short identifier for each transaction in the block. Those identifiers come from SipHash-2-4, keyed from the SHA-256 of the header concatenated with that nonce. The nonce is what makes the scheme safe: identifiers differ for every block and every sender, so nobody can prepare two transactions in advance whose identifiers collide and jam reconstruction across the whole network at once.

Some transactions travel in full rather than by identifier. The coinbase always does, since no peer can have seen it beforehand, and a sender may prefill anything else it suspects the receiver will not have.

Reconstruction then happens on the receiving side. The node hashes every transaction in its own mempool with the same key, looks up each short identifier, and fills the block in. It also keeps a small extra pool of recently evicted and replaced transactions, which catches the ones that dropped out of the mempool moments earlier. Whatever is still missing gets requested by position with getblocktxn, and the sender returns only those transactions in a blocktxn message.

Two things force a fall back to fetching the whole block: too many missing transactions to be worth a round trip, or two different transactions producing the same six byte identifier. Six bytes allows 281 trillion values, so accidental collisions are vanishingly rare, and the deliberate kind is what the nonce rules out.

Announcement comes in two flavours. In low bandwidth mode a peer announces a new block and waits to be asked for it. In high bandwidth mode, which a node enables for at most three peers, the cmpctblock is pushed the instant the block arrives, saving a round trip on the path where propagation speed matters most.

Where you see it

Compact block relay is visible in your own node's log, one line per block, and reading it is the quickest way to judge whether a node is healthy.

Bitcoin Core prints a reconstruction summary naming how many transactions were prefilled, how many came from the mempool, how many came from the extra pool, and how many had to be requested. On a machine that has been running for hours the requested count is usually zero or close to it. On a machine started ten minutes ago the mempool is nearly empty, almost everything has to be fetched, and the first several blocks arrive slowly.

That yields a practical rule for anyone running a node to check their own balance rather than trusting a website: leave it running. A node restarted just before you need it is a node downloading full blocks, and its view of the tip will lag behind a public explorer until the mempool refills.