NEC Laboratories America

Sensing the Physical World | Projects

Safety is often understood only after something goes wrong.

  • A city identifies a dangerous intersection by counting collisions that have already happened.
  • A driver-assistance system reacts to the hazard it can see, even though the greater threat may be hidden around a corner.
  • An environmental program estimates how much debris is in the water from the small fraction that eventually washes ashore.

In each case, the unmet need is the same: a continuous picture of what is actually happening in the physical world, available early enough to act rather than only afterward to explain.

Keyword Tag:  physical world sensing

Cameras alone cannot provide that picture. They see only what is in front of them, under favorable lighting and without obstruction. Yet many of the situations that matter most occur beyond those limits. The collision a vehicle needs to avoid may involve another vehicle hidden around a corner. The traffic conflict a city needs to understand may unfold across an area no single camera can cover. The marine debris an environmental agency wants to measure may be scattered across miles of open water that nobody is continuously watching. New sensing technologies can extend perception beyond these boundaries. Roadside LiDAR, automotive radar, RF sensing, and crowdsourced sensors can detect and track activity across occlusions, darkness, distance, and large physical spaces. Together, they create the possibility of understanding environments that camera-only systems cannot reliably observe.

But infrastructure-scale sensing has a different economic constraint from a laboratory demonstration. Transportation authorities, infrastructure operators, and environmental agencies deploy systems that may need to operate continuously for years. What matters is not simply peak accuracy, but how much physical space can be covered, at what cost, and how much data the system must continuously collect, transmit, and process. A sensing system only becomes useful infrastructure if its economics scale with its coverage. We are building toward a persistent, infrastructure-scale sensing layer for the physical world, one that combines LiDAR, radar, vision, RF, and distributed sensing into a continuous and reliable picture of what is happening.

The goal is to see through the occlusions, darkness, distance, and open spaces that camera-only systems miss, while keeping the cost and data footprint low enough to operate continuously at real-world scale.

Three barriers stand in the way of persistent, infrastructure-scale sensing: bandwidth, geometry, and data scarcity.

  1. First, the sensors that provide the richest view of the physical world also generate enormous amounts of data. A roadside LiDAR can produce point clouds faster than the networks connecting roadside infrastructure can carry them. Compression therefore has to satisfy two requirements at once: preserve small but safety-critical objects such as pedestrians and cyclists, and operate fast enough for real-time use. General-purpose compression methods are poorly suited to this combination. Expanding coverage across multiple LiDARs creates another challenge, requiring independently mounted sensors to be precisely calibrated and synchronized before their observations can be combined.
  2. Second, some of the most important things to sense are not directly visible. Around-corner radar depends on signals that reach a target only after reflecting off surrounding surfaces. Those surfaces have arbitrary and often unknown geometry, making the sensing problem fundamentally different from conventional line-of-sight perception. More broadly, no single sensing modality works reliably in every condition: vision, LiDAR, radar, and RF each fail in different ways. The challenge is to combine them so that their strengths compensate for one another rather than simply combining their errors.
  3. Third, the data needed to build these systems is scarce. High-quality, synchronized multimodal datasets are expensive to collect, rare events are inherently underrepresented, and in domains such as marine monitoring, comprehensive ground-truth datasets may barely exist at all.

Our approach addresses each of these barriers directly. For bandwidth, our work on efficient roadside LiDAR compression makes point-cloud data small enough to move without sacrificing the objects that matter for safety. Its network-aware, real-time successor adapts compression to actual link conditions as they change. Roadside multi-LiDAR fusion then combines observations from multiple units to create a wider and more complete view of the environment. For sensing beyond line of sight, Mosaic turns reflector geometry from an obstacle into a resource, using surrounding surfaces to enable omnidirectional around-corner automotive radar. RoVaR addresses multimodal tracking through deliberate diversity across visual and RF sensing, combining modalities at multiple levels so that one can compensate when another becomes unreliable. Finally, we address the data problem by creating new sources of data rather than waiting for them to exist. StreetAware provides a high-resolution, synchronized multimodal dataset of urban environments. G-Litter uses diffusion models and LLMs to augment scarce and imbalanced marine-litter data. Citizen Science for the Sea takes a different approach, collecting marine and litter observations from instruments already carried by recreational boats and effectively turning a distributed community into a large-scale sensor network.

Our AI systems make persistent sensing practical at a scale that individual sensors cannot achieve: moving high-volume sensor data efficiently, extending perception beyond line of sight, combining modalities that fail differently, and creating the data needed to make the resulting systems work reliably.

Read Related Publications

Roadside Multi-LiDAR Data Fusion for Enhanced Traffic Safety

Roadside LiDAR (Light Detection and Ranging) sensors promise safer and faster traffic management and vehicular operations. However, occlusion and small view angles are significant challenges to widespread use of roadside LiDARs. We consider fusing data from multiple LiDARs at a traffic intersection to

Real-Time Network-Aware Roadside LiDAR Data Compression

LiDAR technology has emerged as a pivotal tool in Intelligent Transportation Systems (ITS), providing unique capabilities that have significantly transformed roadside traffic applications. However, this transformation comes with a distinct challenge: the immense volume of data generated by LiDAR sensors.

G-Litter Marine Litter Dataset Augmentation with Diffusion Models and Large Language Models on GPU Acceleration

Marine litter detection is crucial for environmental monitoring, yet the imbalance in existing datasets limits model performance in identifying various types of waste accurately. This paper presents an efficient data augmentation pipeline that combines generative diffusion models (e.g., Stable Diffusion)

Citizen Science for the Sea with Information Technologies: An Open Platform for Gathering Marine Data and Marine Litter Detection from Leisure Boat Instruments

Data crowdsourcing is an increasingly pervasive and lifestyle-changing technology due to the flywheel effect that results from the interaction between the Internet of Things and Cloud Computing. This paper presents the Citizen Science for the Sea with Information Technologies (C4Sea-IT) framework. It

StreetAware: A High-Resolution Synchronized Multimodal Urban Scene Dataset

Access to high-quality data is an important barrier in the digital analysis of urban settings, including applications within computer vision and urban design. Diverse forms of data collected from sensors in areas of high activity in the urban environment, particularly at street intersections, are valuable

Efficient Compression Method for Roadside LiDAR Data

Roadside LiDAR (Light Detection and Ranging) sensors are recently being explored for intelligent transportation systems aiming at safer and faster traffic management and vehicular operations. A key challenge in such systems is to efficiently transfer massive point-cloud data from the roadside LiDAR devices

RoVaR: Robust Multi-agent Tracking through Dual-layer Diversity in Visual and RF Sensor Fusion

The plethora of sensors in our commodity devices provides a rich substrate for sensor-fused tracking. Yet, today’s solutions are unable to deliver robust and high tracking accuracies across multiple agents in practical, everyday environments – a feature central to the future of immersive and collaborative

Mosaic: Leveraging Diverse Reflector Geometries for Omnidirectional Around-Corner Automotive Radar

A large number of traffic collisions occur as a result of obstructed sight lines, such that even an advanced driver assistance system would be unable to prevent the crash. Recent work has proposed the use of around-the-corner radar systems to detect vehicles, pedestrians, and other road users in these