Teaching AI to Edit Driving Scenes It Has Never Seen with HorizonWeaver

Introduction

This is the problem that NEC Laboratories America’s Media Analytics Department set out to solve with HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes, a paper developed in collaboration with Simon Fraser University. The authors are Mauricio Soroco and Wuyang Chen of Simon Fraser University, along with Francesco Pittaluga, Zaid Tasneem, Abhishek Aich, Bingbing Zhuang, Manmohan Chandraker, and Ziyu Jiang of NEC Laboratories America.

Image Editors Built for The Wrong World

Most image editing tools that follow written instructions were built and trained on photos of everyday objects or artwork, things like a single chair in a room or a landscape painting. Ask one of these tools to change the weather on a city street while keeping every lane marking, traffic sign, and parked car exactly where it belongs, and the results tend to fall apart.

The paper points to three reasons why.

• First, driving scenes are crowded with dozens of interacting objects. Hence, an editor must make small, precise changes to individual vehicles or pedestrians while also handling changes to the whole scene, such as switching from day to night, without one kind of edit disturbing the other.
• Second, an editor needs to preserve everything the instruction did not ask it to touch, since safety depends on details like lane geometry and traffic signs staying intact, while still following complex instructions with several conditions at once.
• Third, the handful of driving datasets available today were mostly recorded in a small number of cities and weather patterns, so editors trained on them struggle the moment they see a new environment.

On a benchmark built specifically to test scenes the model had never encountered, existing editors such as Qwen-Image, UltraEdit, BAGEL, and OmniGen2 all showed measurably higher error rates than the approach the team introduces in this paper.

Masks That Speak Instructions

HorizonWeaver addresses these three problems with contributions to the data, the model, and the training process. On the data side, the team built a paired dataset of 255,000 images across 13 categories of driving scene edits, such as adding a pedestrian or changing the weather, drawn from three real-world driving datasets called Boreas, nuScenes, and Argoverse2. Rather than editing a single image and hoping the result looks realistic, the team paired real photographs taken from nearly the same camera position on different days, so the model can learn what changes between a clear afternoon and a snowy one.

On the model side, the paper introduces LangMasks, a way of attaching short text instructions directly to specific pixel regions in an image, like circling a car in a photo and writing a note next to it that says replace this with a white van. This lets the model make several precise, localized edits inside a single busy scene without confusing one instruction for another.

On the training side, the researchers designed a pair of loss functions, mathematical penalties used to guide how a model learns, that reward the model for keeping untouched parts of the scene consistent while still following the instructions closely for the parts it is asked to change.

Numbers That Hold Up

On a set of scenes drawn from nuPlan, a dataset the model never saw during training, HorizonWeaver reduced pixel-level error, measured as L1 distance, by 45.5 percent compared with the strongest competing method, while improving CLIP similarity by 12.1 percent and DINO similarity by 24.1 percent, two measures of how closely an edited image matches its intended content and style.

In a study where 26 people reviewed 409 edited images side by side with four leading alternatives, including Qwen-Image, UltraEdit, BAGEL, and OmniGen2, HorizonWeaver’s edits were chosen as the best match to the instructions 61.1 percent of the time, compared with 14.7 percent for the next best method.

The team also tested whether the added scene variety helps downstream driving software. After training a bird’s-eye view map segmentation model, which identifies road features like crosswalks and lane dividers from overhead-style imagery, on real images augmented with HorizonWeaver-edited scenes, the mean accuracy of that model rose from 0.1859 to 0.2477, an improvement of 33 percent.

That Minneapolis delivery truck the safety engineer needed but never filmed is exactly the kind of scene HorizonWeaver is built to generate: a real Phoenix highway edited into a snow-covered street, with the truck, lane markings, and traffic signs all staying where they belong.

Where This Fits Today

Autonomous vehicle developers and automotive suppliers can use this kind of scene editing to stress test perception and planning software against weather, lighting, and traffic conditions that are too rare or too dangerous to capture on public roads. Simulation and digital twin companies can go further, using the same techniques to multiply a limited set of recorded drives into a much larger library of test scenarios, cutting down on how much new footage needs to be filmed in the first place. Government transportation safety agencies and academic researchers, meanwhile, have less use for the editing tool itself and more for what it produces: the dataset and evaluation benchmarks give them a shared, repeatable way to compare how different perception systems handle unfamiliar environments.

Driving scenes are some of the hardest environments to edit convincingly because a single mistake, like erasing a stop sign or misplacing a lane, is not just a visual flaw; it is a safety issue, said Francesco Pittaluga, Senior Researcher at NEC Laboratories America. We built HorizonWeaver so that the model understands the difference between the objects that safely can change and the ones that never should.

The authors note that HorizonWeaver still relies on instructions generated by large language models and filtering performed by vision language models, both of which can introduce occasional errors or unrealistic edits. They point to human-in-the-loop verification as a direction for future work, alongside continued efforts to broaden the range of driving environments and edit types the model can handle.