When Video AI Gets Physics Wrong, the Consequences Are Real

About the Reasearch

A new paper titled “PhyCo: Learning Controllable Physical Priors for Generative Motion“ addresses this gap directly. The work comes from our Media Analytics department, in collaboration with Carnegie Mellon University and the University of California San Diego. The research team includes Sriram Narayanan (NEC Laboratories America Intern and Carnegie Mellon University), Ziyu Jiang (Researcher at NEC Laboratories America), Professor Srinivasa G. Narasimhan (Carnegie Mellon University), and Manmohan Chandraker (Media Analytics Department Head at NEC Laboratories America and Professor at UC San Diego).

The Problem: Video Models Look Real but Act Wrong

Modern video diffusion models, which generate video from text or image prompts using a process of iterative noise removal, have made remarkable progress on visual realism. They can render convincing lighting, textures, and motion continuity. But they consistently fail at physics. Objects drift rather than slide. Collisions produce no rebound. Soft materials do not deform when they should. These are not edge cases. They are systematic failures that emerge because current models are trained to look right, not to behave correctly.

The deeper problem is control. Even when a model occasionally produces physically plausible output, there is no reliable way to specify how stiff, how bouncy, or how friction-prone a surface or object should be. Practitioners cannot dial in a friction coefficient or a restitution value and expect the generated video to respect it. That missing layer of control is precisely what PhyCo is designed to provide.

The Solution: Teaching a Model to Understand Physics Through Simulation

The PhyCo framework introduces physically grounded control into video generation through three interlocking components. The first is a large-scale synthetic dataset of more than 100,000 photorealistic simulation videos. The team built this dataset using the Kubric simulation framework with PyBullet for physics and Blender for rendering, spanning six scenarios: a brick sliding on a flat plane, a ball rebounding off a wall, a vertically bouncing ball, a soft ball dropping under gravity, an object impacting a deformable body, and multiple balls colliding on a pool table. Across all scenarios, four key physical parameters vary systematically: friction (resistance to sliding), restitution (bounciness after impact), deformation (how much a material compresses or stretches), and applied force (the magnitude and direction of a push or pull).

The second component is physics-supervised fine-tuning. Rather than retraining a video model from scratch, the team fine-tunes a pretrained video diffusion model, Cosmos-Predict2-2B, using a ControlNet architecture. A ControlNet is a lightweight conditioning layer that injects additional input signals into an existing neural network without modifying the core model. In this case, the ControlNet receives pixel-aligned physical property maps, spatial overlays that encode the friction level, restitution coefficient, deformation parameters, and force direction directly onto the video frame, pixel by pixel. The model learns to associate these maps with the corresponding physical behavior in the simulation.

The third component is a feedback loop powered by a vision-language model (VLM). This AI system can analyze images or video and answer questions about them in natural language. After the diffusion model generates a video, a fine-tuned version of Qwen2.5-VL-3B evaluates it by answering targeted physics questions, such as: “Does the ball bounce with the specified level of restitution?” Does the object move in the direction of the applied force?

The answers produce a reward signal that the model uses to improve its physical accuracy over time, without requiring any additional labeled training data.

The Results: Measurable Gains in Physical Realism

The team evaluated PhyCo on the Physics-IQ benchmark. This standardized evaluation suite measures how well generated videos match real-world physical behavior across five domains: solid mechanics, fluid dynamics, optics, magnetism, and thermodynamics. The benchmark compares the timing and spatial alignment of key actions in generated videos against reference recordings of real physical events.

PhyCo, with both ControlNet conditioning and VLM-guided reward optimization, achieved an overall Physics-IQ score of 43.6, compared to 34.6 for VLIPP, the prior leading method, and 27.7 for the base Cosmos-Predict2-2B model used as the starting point. In solid mechanics, the domain most directly relevant to friction, restitution, and deformation, PhyCo reached 53.1, compared to 42.3 for VLIPP. These are substantial gains on a benchmark designed to be difficult for current video models. The fine-tuned VLM evaluator also achieved 84.8 percent accuracy in predicting physical property values from generated video, and the model’s mean angular error in force direction fell to 15.2 degrees, compared to 40.5 degrees for the prior Force-Prompting method.

Importantly, the model generalizes beyond the scenarios on which it was trained. A model trained on simple bouncing-ball simulations can produce physically coherent video of a person jumping on a trampoline, with low restitution settings correctly yielding no rebound after impact. Training on blocks sliding across flat surfaces transfers naturally to more complex object-and-surface configurations.

Real-World Applications

Product design and materials engineering are immediate beneficiaries. Engineering firms that need to prototype how materials behave under stress, or simulate how consumer products respond to impact, currently rely on explicit physics solvers that require detailed 3D geometry and material specifications as inputs. PhyCo offers a complementary path: generating photorealistic video previews of physical behavior from a single image and a set of property parameters, with no simulator required during inference. Automotive companies, aerospace contractors, and industrial manufacturers could use this to accelerate early-stage design review.

Robotics and embodied AI training pipelines represent a second high-value application. Systems designed to teach robots to manipulate objects require large volumes of physically plausible interaction video data. Generating that data synthetically is far cheaper than collecting it in the real world, but only if the generated physics is trustworthy. PhyCo’s ability to control friction, deformation, and applied force parameters programmatically makes it well suited to generating diverse, physically labeled training data for robotic grasping, pushing, and assembly tasks. Research labs and robotics companies building foundation models for physical AI would benefit directly.

“Most video generation models learn to look physically plausible, but they have no internal representation of what friction or bounciness actually means. With PhyCo, we wanted to give the model a vocabulary for physics, not just appearance. What excites me most is that the model learns this vocabulary from simulation and then applies it to real-world scenes it has never seen. That generalization is what makes it practically useful, not just a benchmark result.”

— Ziyu Jiang, NEC Laboratories America