The standard approach is to apply modern computer vision models to camera feeds and assume that strong benchmark performance will translate into the real world. In practice, it often does not work. A model’s accuracy on a benchmark and its accuracy on a street at dusk in the rain can be very different. The model itself is only part of the reason. Everything between the physical scene and the model affects the final result: camera settings, image quality, encoding, network conditions, available compute, and scheduling decisions. A strong model running on a poorly configured pipeline can still produce poor analytics. For industries deploying video analytics at scale (like public safety, transportation, retail, healthcare, and manufacturing), this gap determines whether a system is trusted and ultimately adopted. Systems may be purchased based on benchmark accuracy, but their long-term value depends on the accuracy they deliver in the field. Yet vendors have limited ability to control that performance once conditions change.

We are building a self-optimizing video analytics pipeline. Rather than treating cameras, networks, compute, and AI models as independent components, the system manages them as a single end-to-end pipeline. It observes its own analytics performance, identifies where quality is being lost, and continuously adjusts camera settings, resource allocation, and scheduling as conditions change. The goal is video analytics that maintains its own quality in the real world, while adapting to changing scenes, workloads, networks, and resources without requiring a human to constantly retune the system.

The central research challenge is that real-world accuracy cannot be directly observed at runtime. A live video stream comes without ground truth, so the system cannot simply measure its accuracy and correct itself when performance declines. The problem starts at the camera. Camera parameters were designed primarily to produce images that look good to people, not images that make objects and features easy for AI models to recognize. As a result, the settings a camera selects automatically can be poorly suited for analytics—and model retraining alone cannot fix information that was lost before the video ever reached the model. The causes of performance degradation are also deeply interconnected. Video is not a sequence of independent images: scenes evolve over time, resource allocations that work well under one condition become inefficient as the scene changes, and multiple analytics pipelines compete for shared compute and network capacity. Maintaining accuracy therefore requires continuously adapting the entire camera-to-insight pipeline.

Our approach is to build proxies for quality and closed-loop controls where ground truth is unavailable. AQuA provides a lightweight estimate of whether a frame is useful for analytics without first running the full model, while our work on understanding fluctuations in video analytics accuracy identifies the underlying causes of performance variation rather than treating them as random noise. With those signals in place, CAMTUNER and APT close the control loop around the camera itself. They use reinforcement learning to adjust camera parameters in real time based on perceptual and analytics-driven objectives rather than human visual preferences. Elixir extends this idea to streams shared by multiple analytics tasks, optimizing data quality across several applications at once.

The next set of control loops optimizes the infrastructure around the model. Magic-Pipe continuously adjusts pipeline configurations and resource allocation as conditions change. EdgeSync schedules edge-assisted analytics with awareness of contention on shared networks, while SimCache exploits similarity between scenes to avoid repeated vision-language inference on nearly identical content. Finally, TrafficLens and our work on few-shot video classification broaden what these pipelines can understand. TrafficLens extends analytics across multiple cameras, while few-shot classification enables recognition of new classes even when very little labeled training data is available.

Our AI systems move video analytics from a fixed pipeline that degrades as the real-world changes to a closed-loop system that observes, adapts, and continuously optimizes itself, from the camera capturing the scene to the infrastructure and models producing the final insight.