The system had passed everything. Every firmware unit test, every bench check on the hardware side, every software integration test in the lab. Then we connected the two halves on the real target for the first time, and it failed within seconds. Not a crash with a stack trace. Something worse: it ran, but the numbers were quietly wrong, in a way none of the tests were built to catch.
The cause was not in anyone's code. It was in the half-sentence nobody had written down: what one signal actually meant, and when it was valid. The firmware team and the software team each had a reasonable interpretation. They were different. Everything each team owned was correct, and the system still failed.
That is the pattern I have watched repeat for fifteen years.
Risk doesn't live inside the boxes.
It lives in the seams between them.
01 The failure modes live between the boxes, not inside them
Every discipline optimizes its own box. A subsystem tested in isolation is correct by its own definition, and it usually is. The interface is where two definitions meet, and where they can silently disagree: a value in raw counts on one side and engineering units on the other, a message rate one side assumes and the other never promises, an initialization order that only matters on real hardware, a "ready" flag that means "powered" to one team and "calibrated" to the other.
None of those is a bug in a component. Each side passed its own tests because each side tested its own understanding. So when integration goes wrong, the instinct to re-check the components is usually wasted effort. Look at the boundary first. The components were tested. The boundary almost never was.
02 Nobody owns the interface by default
Org charts own boxes. Teams, repositories, and budgets all map cleanly to subsystems. The interface between two subsystems maps to nobody. It is an orphan.
An orphaned interface quietly collects unstated assumptions from both sides, because each side assumes the other has it handled. The most reliable predictor of integration pain I know is a boundary with no name attached to it. The fix is not technical, it is organizational: assign interface ownership as deliberately as you assign a component owner. Whoever owns the contract owns the risk, and someone has to, or it owns you.
03 The boundary is where assumptions go to hide
Hardware assumes something. Software assumes something. Both are reasonable, both are undocumented, and it all works, right up until a condition changes: heavier load, a colder start, a faster message rate, a new build that shifts timing by a few milliseconds. Then the two assumptions stop agreeing, and the system does something nobody predicted, at the worst possible moment, because the failure only shows up under the condition nobody wrote a test for.
The remedy is boring and it works: make the implicit explicit. A written interface contract, covering units, ranges, rates, timing, coordinate frames, and above all the error behavior. What happens when the signal is late, absent, or out of range? That contract is not documentation. It is the risk register for the boundary.
04 How I de-risk a boundary now
- Name the owner of every interface, not just every component.
- Write the contract: units, ranges, rates, timing, frames, and error semantics.
- Test the boundary directly, not just the boxes on either side.
- For each signal, ask the uncomfortable question: what happens when it is late, missing, or wrong?
- Assume the two sides interpret it differently until proven otherwise.
A system doesn't fail at the components you tested. It fails at the seams you assumed. The highest-leverage move I know in complex delivery is unglamorous: name the boundaries, write down what crosses them, and decide who owns the answer when the two sides disagree.