Real-Time Computer Vision on Mobile with MediaPipe

Real-time body tracking that has to run smoothly on a mid-range phone, inside a Unity app, is a very different problem from body tracking running on a server GPU. Cutting processing latency by 40% came down to a handful of deliberate trade-offs.

Pick the lightest model that meets the bar

MediaPipe ships model complexity tiers for pose and landmark detection. It's tempting to default to the most accurate model — but on mid-range hardware, the accuracy gain over the "lite" tier rarely matters for the actual use case, while the latency cost is significant. Profiling the real target devices, not a flagship phone, is what surfaced this.

Where the frames actually go

The pipeline runs MediaPipe's landmark detection on-device, then streams normalized landmark coordinates — not raw video — into Unity for rendering and interaction logic. Keeping video frames out of Unity's managed memory entirely and passing only lightweight landmark arrays across the native/managed boundary removed a marshaling bottleneck that was invisible in profiling until it was isolated with targeted native timers.

Other latency wins

• Frame skipping with interpolation — running detection at a lower frame rate and interpolating landmark positions between detections stayed visually smooth while cutting inference calls significantly.
• GPU delegate where available — MediaPipe's GPU inference path made a meaningful difference on devices that supported it, with a CPU fallback for those that didn't.
• Avoiding per-frame allocations — reusing buffers instead of allocating new arrays every frame reduced GC pauses that showed up as visible stutter, not just raw latency.

The overall lesson: on mobile, the model is rarely the only bottleneck — the data path around it usually costs just as much, and it's worth profiling before assuming a smaller model is the answer.