97% Test Coverage: CI Pipelines for ML Services

A CI pipeline that reached 97% test coverage and cut production errors by 95% wasn't the result of chasing a coverage number — it was the result of testing the parts of an ML platform that actually break in production, which are rarely the model weights themselves.

What actually broke, historically

Looking back at real incidents before this pipeline existed, almost none were "the model gave a wrong prediction." They were: a schema mismatch between a client and a model's input contract, a silent config change that pointed a service at the wrong model version, a serialization edge case in a specific input shape, or a resource limit that only showed up under real concurrency. The test suite was built to target exactly these failure modes.

The pipeline shape

• Contract tests — every model service's request/response schema is tested against real client payloads on every PR, so a breaking change is caught before merge, not in production.
• Integration tests against containerized services — spinning up the actual Docker images in CI, not mocks, catches the class of bug that only shows up when the real container boots with real config.
• Golden-output regression tests — a fixed set of inputs with expected output ranges runs on every model or pipeline change, catching silent regressions in preprocessing or postprocessing logic, not just crashes.
• Load smoke tests — a short burst of concurrent requests against a staging deployment, gating rollout on latency and error-rate thresholds rather than just "tests passed."

Why 97% coverage mattered less than it sounds

Coverage was a useful forcing function — it surfaced untested edge cases and dead code — but it was a byproduct of writing tests aimed at real failure modes, not the goal itself. The 95% drop in production errors came from testing the boundaries between services (schemas, configs, versions) far more than from testing individual functions in isolation.