Neural Rendering is an AI-driven technique that synthesizes realistic images, videos, or 3D scenes by learning from data rather than relying on traditional computer graphics pipelines. It leverages deep learning models, such as generative adversarial networks (GANs) and neural radiance fields (NeRFs), to generate high-fidelity visual content. This technology is widely used in applications like virtual reality, gaming, and film production to create lifelike digital environments and characters. Neural rendering can also enable real-time scene manipulation, allowing users to modify lighting, textures, and perspectives dynamically. As the field advances, it continues to blur the line between synthetic and real-world visuals, revolutionizing content creation across industries.

Posts

AutoScape: Geometry-Consistent Long-Horizon Scene Generation

/in /by

This paper proposes AutoScape, a long-horizon driving scene generation framework. At its core is a novel RGB-D diffusion model that iteratively generates sparse, geometrically consistent keyframes, serving as reliable anchors for the scenes appearance and geometry. To maintain long-range geometric consistency, the model 1) jointly handles image and depth in a shared latent space, 2) explicitly conditions on the existing scene geometry (i.e., rendered point clouds) from previously generated keyframes, and 3) steers the sampling process with a warp-consistent guidance. Given high-quality RGB-D keyframes, a video diffusion model then interpolates between them to produce dense nd coherent video frames. AutoScape generates realistic and geometrically consistent driving videos of over 20 seconds, improving the long-horizon FID and FVD scores over the prior state-of-the-art by 48.6% and 43.0%, respectively.

Read more

LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

/in /by

Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural radiance fields (NeRFs) may allow better scalability through the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely collinear camera motions and sparser samplings at higher speeds. On the other hand, the application often demands rendering from camera views that deviate from the inputs to accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes. First, our framework learns a geometric scene representation from Lidar, which are fused with the implicit grid-based representation for radiance decoding, thereby supplying strongergeometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views from Lidar points for further improvement. Our insights translate to largely improved novel view synthesis under real driving scenes.

Read more