A team ships a release. The test suite is green. Coverage is 87%. CI has been clean for weeks. Two weeks after the release, a customer reports the system silently drops events when their upstream queue is slow, a failure mode nobody has tests for because nobody had ever thought to test it. The team is technically right that "we tested it." They are also technically wrong about what that means. This is the gap where most validation discussions go wrong. "Tested" is a binary signal (green/red, pass/fail). "Validated" is a different operation: it answers a specific question about whether the system behaves the way you claim under the conditions that will actually matter.
Tests answer questions. Most test suites don't have a list of the questions,
they have a list of the implementations that happen to pass.
01 "Tested" is not "Validated"
The two words look interchangeable. They are not.
"Tested" means: tests ran, tests passed. "Validated" means: the team has evidence, under representative conditions, that the system behaves the way they claim it does.
Tests can pass for many reasons. Some have nothing to do with the system being correct: the test asserts behaviour the implementation was written to produce, with no independent definition of what "correct" means. The test exercises the happy path of a function but never the conditions under which that function actually fails in production. The test runs against mocked dependencies whose behaviour does not match reality, the test was written after the implementation, so the test inherits the implementation's blind spots.
A green test suite tells you that the tests pass. Whether that pass means the system is validated depends on what questions the tests were designed to answer, and whether anyone wrote those questions down.
Pick three tests in your codebase at random. For each, in one sentence, state the specific question the test is designed to answer. If you can't, those tests are passing without validating.
02 The accreted-artifact problem
Most test suites are not designed. They accrete.
The pattern is recognisable. Sprint 1: the engineer writes the feature, adds a few tests for the obvious cases. Sprint 5: a bug surfaces, someone adds a regression test for the specific scenario. Sprint 12: a refactor breaks something nobody anticipated, someone adds a smoke test. Sprint 30: the team is on its third architecture, and the test suite has 1,400 tests, of which the original 8 are still meaningful, ~200 are obsolete but still green, and the rest are a mix of accumulated reactions to past surprises.
That test suite passes. It looks comprehensive. It is comprehensive about what has gone wrong in the past. It is not necessarily comprehensive about what is most likely to go wrong next.
This is why coverage numbers are a misleading proxy for validation quality. 87% coverage means 87% of lines have a test passing somewhere. It does not mean those tests answer questions about the system's actual behaviour under the conditions that matter.
03 The shadow contract
Every test makes assumptions about the world the code runs in. Most of those assumptions are invisible. They are written into the test setup, the mocks, the fixtures, the test data, and together they form a shadow contract: the set of conditions under which the tests pass.
Production has its own contract: the actual conditions the system encounters in the wild. When the shadow contract matches the production contract, tests passing implies the system works. When they diverge, tests passing means nothing.
Examples of divergence: tests use small fixtures while production has 100x the data volume; tests mock external services with cooperative responses while production has hostile or absent responses; tests run on a clean clock while production has clock skew, retries, race conditions; tests assume happy ordering while production has out-of-order, duplicated, dropped events.
The shadow contract drifts from the production contract continuously. Nobody writes it down. Nobody reviews it. When integration day surfaces a behaviour the tests never exercised, the team's reaction is usually "we should have tested that" which is true, but the deeper question is: why didn't the test suite already know it needed to?
A test suite that has not been compared, in detail, to the production conditions it claims to validate is not a validation artifact. It is a regression suite, useful for catching past mistakes, not for proving new claims. The two are not the same, and treating them as the same is where the largest validation gaps form.
04 Three checks that expose validation gaps
You can run three checks that surface the gap between what your tests claim and what your system actually does. None requires new tooling.
None of these requires new tooling. All three surface the validation gaps that pass-rate dashboards cannot.
"We tested it" answers a different question than "it works." Tests pass when the implementation matches the test's assumptions. Whether those assumptions match the conditions the system will actually face is a separate question — and one most teams never write down. The gap between the shadow contract (what tests assume) and the production contract (what production delivers) is where validation fails silently.
Validation is not a count of tests. It is a structured argument about why the team should believe its own claims about the system. Tests are the evidence layer underneath that argument — but only when each test answers a specific, documented question about the system's behaviour. The SYS Readiness Check is built to surface where that argument is structurally weakest, across all five dimensions: definition clarity, system structure, validation readiness, change exposure, and execution control.