MLOps at Scale: Serving 20M Requests a Day

Serving 30+ ML models behind a single platform, peaking at 20 million requests a day with 99.5% uptime, is less about any one clever optimization and more about a serving architecture that doesn't fight you as traffic grows. Here's how that architecture was put together on FastAPI, Docker and Kubernetes.

Isolate models, share infrastructure

Each model gets its own containerized service with its own resource profile — a lightweight classifier doesn't need the same GPU allocation as an image generation model. FastAPI fronts every service with a consistent contract (health checks, structured logging, request/response schemas via Pydantic), so onboarding a new model means implementing one interface, not reinventing the serving layer.

Autoscaling that matches real traffic shape

Horizontal Pod Autoscaling on Kubernetes, tuned on request latency and queue depth rather than raw CPU, handled the daily traffic curve without over-provisioning at 3am. GPU-backed pods use a separate, more conservative scaling policy since cold-starting a GPU workload is expensive — a small always-warm baseline plus fast scale-out absorbed spikes without wasting spend during quiet hours.

What actually kept uptime at 99.5%

• Readiness probes tied to real model-load state, not just "process is running" — traffic never hits a pod mid-warmup.
• Circuit breakers per downstream model so one misbehaving model can't cascade into a platform-wide outage.
• Canary rollouts for every model version bump, with automatic rollback on error-rate regression.
• Structured, sampled logging that stays cheap at 20M requests/day but still gives enough signal to debug a specific failed request.

None of this is exotic. The pattern that held up under real load was consistency — every model behind the same contract, the same health checks and the same rollout process — so operational effort didn't scale linearly with the number of models in production.