← Back to Insights

Software Quality & Confidence

Why Test Coverage Is Not Confidence

What a banking platform's Transaction Ledger can teach us about coverage, dependency behavior, and the evidence needed to justify a release.

Editorial illustration of a sleeping man whose blanket leaves his feet exposed, a metaphor for gaps in test coverage.
Editorial illustration · a visual metaphor

We've spent decades measuring how much software we test. Far less time asking whether those tests give us a good reason to trust what we release.

I understand the appeal of coverage. It gives an abstract activity a visible shape. We can locate unexecuted code, challenge missing scenarios, and notice when a change escapes the checks we expected to protect it. Those are useful engineering conversations. The problem begins when a description of test execution quietly becomes permission to stop asking questions.

Confidence is a judgment about a particular decision under particular conditions. Coverage describes part of the evidence available for that judgment. Treating the two as interchangeable hides the assumptions on which a release depends.

Three questions that deserve separate answers

Functional coverage asks which specified behaviors we have checked. Can a payment be accepted, rejected, cancelled, or retried according to its contract? Code coverage asks which instrumented statements or branches were executed during those checks. Risk relevance asks whether the observations address the failures that matter for the change we are considering.

These questions overlap without collapsing into one another. A test can execute a branch while making a weak assertion. A carefully specified functional test can establish a critical invariant while observing little about resource contention. Neither result is worthless. Each becomes misleading when it is used to support a broader claim than its design permits.

I prefer to ask what would have to be true for our current evidence to remain useful. That invites discussion about dependency state, traffic, timing, configuration, and what the test environment deliberately leaves out.

An illustrative case: Payment Orchestration and Transaction Ledger

Consider a cloud-native banking platform with two services: Payment Orchestration coordinates a payment, while Transaction Ledger records its financial state. This is an illustrative scenario, not a reported customer incident or a benchmark. No outcomes or numerical improvements here should be read as production evidence.

An engineer improves idempotency in Payment Orchestration. Requests sharing an idempotency key should represent one logical payment, even when a client retries. The implementation correctly preserves that invariant. The relevant functional tests pass, including duplicate submissions and recovery after a response is lost.

Suppose the improvement also changes the interaction with Transaction Ledger. Before deciding whether to reuse an existing result, the orchestration service performs an additional ledger lookup. A timeout can initiate another attempt. The logical payment remains unique, yet the number and timing of calls to the dependency can change.

Now consider a ledger already experiencing degraded latency or intermittent timeouts. More requests remain outstanding while callers wait. Retries add work precisely when the dependency has less room to absorb it. Depending on the actual concurrency controls and retry policy, this can increase pressure across the boundary without creating a duplicate payment.

Nothing in this description requires the functional assertion to be wrong. Correctness of the payment invariant and acceptable behavior under dependency stress are different properties. A stub returning immediately can help test the first while telling us very little about the second.

The tests aren't lying. We're asking them to answer a question they were never designed to answer.

What the test establishes; what the system still owes us

The passing tests establish that the checked inputs produced the expected outcomes in the conditions exercised. They do not establish a universal safety claim about the service graph. They might not model concurrent retries, exhausted connection pools, delayed acknowledgments, or recovery after the dependency becomes healthy again.

This is where I want a more precise conversation, rather than an argument about whether the coverage percentage is high enough. Which interaction changed? Which resource could become constrained? Which failure would be consequential? Which observation could challenge our belief that the change is safe?

The AWS Builders' Library discussion of timeouts, retries, and backoff with jitter (opens in a new tab) explains why retries require care around overloaded dependencies. That is relevant engineering guidance, not proof that this illustrative platform will fail. We still need evidence from the system and conditions we intend to evaluate.

Runtime signals make the missing context visible

For the ledger interaction, useful observations could include dependency latency distributions, timeout frequency, outstanding requests, retry attempts per logical operation, and saturation of the resources involved. Traces can help connect the orchestration change to the calls it makes. Their usefulness depends on instrumentation, sampling, and whether the observed requests represent the behavior under discussion.

I would compare those observations before and after the change under a defined workload, introduce controlled dependency degradation, and inspect recovery. I would also preserve assertions about payment uniqueness and ledger state. The aim is to connect functional correctness with operational behavior, not substitute one for the other.

A signal is still an observation with limits. A stable average can conceal a problematic tail. A missing trace can be a sampling artifact. A staging workload can differ materially from production. Better evidence makes those limitations discussable; it does not abolish them.

Turn confidence into an explicit argument

Before approving this change, I would want a short, reviewable account of the decision:

  • The invariant protected by functional tests: one logical payment for the same idempotency key.
  • The changed dependency interaction: additional ledger work or different retry behavior.
  • The operational conditions examined: representative concurrency, dependency degradation, and recovery.
  • The remaining uncertainty: conditions we could not reproduce, observe, or confidently bound.
  • The release safeguards: an owner, a limited rollout, and explicit conditions for pausing or reversing it.

Those statements do not need a universal confidence score. They need enough specificity that another engineer can challenge them. The acceptable residual uncertainty will differ between a low-impact interface change and a payment path whose failure can affect customers' money.

This is the intellectual starting point of Software Signal Engineering for me: interpreting evidence in context before using it to justify a decision. It is an approach to engineering judgment, not a claim that a new label scientifically validates a release.

Keep coverage. Stop asking it to be certainty.

Traditional testing remains indispensable. Without clear contracts, meaningful assertions, and repeatable checks, operational observations become harder to interpret. A test suite can catch regressions efficiently and make changes possible that would otherwise be too risky to attempt.

But a release conversation should end with a statement about the change, its dependencies, and the evidence supporting the decision. It should not end merely because a familiar indicator is green. We earn confidence by explaining what we know, how we know it, and what could still prove us wrong.

Your privacy choices

With your permission, Google Analytics measures page visits, outbound links, and successful contact or book confirmations. Advertising remains disabled. Contact and book signup work without analytics. Read the privacy policy.