PhyCo: Learning Controllable Physical Priors for Generative Motion

/in /by

Publication Date: 6/3/2026

Event: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026)

Reference: pp. 41892-41902, 2026

Authors: Sriram Narayanan, NEC Laboratories America, Inc., Carnegie Mellon University; Ziyu Jiang, NEC Laboratories America, Inc.; Srinivasa Narasimhan, Carnegie Mellon University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes–without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.

Publication Link: https://openaccess.thecvf.com/content/CVPR2026/html/Narayanan_PhyCo_Learning_Controllable_Physical_Priors_for_Generative_Motion_CVPR_2026_paper.html