Writing the protocol twice

The bug that agrees with its test

Testing normally asks: does the code do what I expect? You write the expectation as an assertion and the machine checks it. This finds mistakes in the code and misses mistakes in the expectation.

If you misread the specification — took a damped value where the formula meant an undamped one, assumed an operation covers one unit where it can cover many — you will write the test with the same misreading. The test passes. Coverage is complete. The bug is still there, and it is invisible from inside.

Differential testing

The way out is to build a second implementation that does not share the first one’s assumptions, then run both on the same inputs and compare. Where they disagree, at least one is wrong — and you have to go back to the specification to find out which.

Independence is the whole point, and it is easy to lose:

• Write it from the specification, not from the code. Reading the code first imports its assumptions, and you get an expensive translation rather than a check.
• Use a different language. Different arithmetic, different overflow behaviour, different idioms — the mismatches this produces are informative rather than annoying.
• Let it be slow. The reference is not going to be deployed. Clarity beats efficiency; write the formula the way the document states it.
• Ideally, a different person. Not always possible. Where it is not, put time between the two — a specification read afresh is read differently.

What it actually finds

Two examples from this project, both found by comparison and neither by tests:

• A mixed pair of values. The headroom calculation in one safety check used undamped figures while the quantity it was protecting used the damper. Every unit test passed, because the tests computed the same way. The reference, written straight from the formula, produced a different number on a small pool — and on a small pool the safety threshold could be breached.
• An undefined case. The issue price was specified for a unit, and operations can cover very many units at once. The contract and the tests both silently assumed the small case. Read literally, a large deposit would have claimed a whole phase at the opening price.

Neither is exotic. Both are the ordinary result of one mind writing both sides.

Where the reference stays useful

It is not a one-off exercise. Once it exists, the reference keeps paying:

• Property tests can throw thousands of random inputs at both and compare — a fuzzer with an oracle attached, rather than a fuzzer looking only for crashes.
• Changes get checked against it, so a refactor that quietly alters behaviour shows up as a disagreement.
• It is readable by people who do not read Solidity, which makes review possible for a wider set of eyes.
• When the two disagree and the specification is ambiguous, that is a defect in the specification — found before anyone deployed anything.

The related trick: attack your own tests

A second question is whether the tests are looking at all. Mutation testing answers it: inject a deliberate defect into the compiled code — flip a comparison, change a constant — and run the suite. If it still passes, that defect is in a region nothing observes.

It is uncomfortable in a useful way. Coverage says which lines ran; mutation says whether anything would have noticed if those lines were wrong. The two are not the same, and only the second is a property of the tests.

What it costs

A reference implementation of a protocol of this size is days, not months, precisely because it is allowed to be slow and plain. Set against the cost of finding a threshold bug after deployment, the arithmetic is not close.