Perception error model

Short description: Autonomous vehicle simulation method

In autonomous vehicle testing, a perception error model (PEM) is an approach to simulating the behaviour of sensing and perception systems by modeling the errors they produce rather than simulating the underlying sensor physics. This differs from high-fidelity sensor simulation techniques, where one wishes to generate synthetic sensor signals that the actual perception algorithms would process. Perception error models are sometimes referred to as surrogate models for perception systems, because they approximate the output of computationally expensive object detectors at a fraction of the computational cost.

The primary motivation for PEMs is to enable efficient virtual testing of autonomous vehicle driving policies. Since the failure modes of perception systems have significant impact on downstream planning and control, testing these systems requires capturing the dependencies between perception errors and vehicle behaviour. PEMs provide a computationally efficient alternative to running full physics-based sensor simulations with actual perception algorithms.

Definition and design

Formal definition

A formal definition of perception error models was proposed as an approximation of the combined function of sensing subsystem S and perception subsystem P:

$\text{PEM}(W)\approx S\mathrm{\&}P(W)=\tilde{W}=W+E$

where W denotes the ground truth world state consisting of surrounding objects, $\tilde{W}$ denotes the perceived world, and E represents the perception error. The PEM receives the ground truth world W and returns the perceived world $\tilde{W}$.

The driving policy DP then generates a response R by analyzing the perceived world:

$R=DP(\tilde{W})=DP(W+E)$

This definition does not depend on specific sensor types and provides a standard interface for integration in simulation pipelines.

Design considerations

Four key factors have been identified that affect the manifestation of perception errors in autonomous vehicles, which should be considered when designing a PEM.

Positional aspects: The relative position of objects with respect to the ego-vehicle affects the quality of perception. Sensors have limited field of view (FoV), including limited range and blind spots. Signal strength and resolution degrade with distance, and perception systems typically perform better in areas where multiple sensor fields of view overlap.

Parameter inter-dependencies: Object parameters such as class and size affect error manifestation. Larger objects are more likely to be detected, slow-moving vehicles are easier to track, and classification errors may affect size estimation. Parameters not directly relevant to driving policy may also be important, such as object colour or material affecting LiDAR or radar performance.

Occlusion: Objects in the scene may influence detection of other objects. Large vehicles such as trucks may occlude smaller objects such as cars or cyclists. Occlusion significantly impacts perception reliability, and cumulative statistics should be reported separately for different occlusion levels.

Temporal aspects: Objects move through the scene, causing previously occluded objects to become visible and vice versa. Algorithm uncertainties and filtering techniques depend on their previous state, so errors evolve over time and should be modeled as time series using dynamical models.

Applications

Data-driven perception error models

A generalized data-driven approach to PEM construction was proposed using the Apollo autonomous driving software and the nuScenes public dataset. This approach models the Sensing and Perception system as a whole using Hidden Markov Models, partitioned based on spatial location around the ego-vehicle and occlusion levels. The key contribution was demonstrating PEM integration into a simulation pipeline, enabling analysis of how perception errors affect autonomous vehicle safety. Similarly, neural networks have been tested as efficient surrogates in the CARLA simulator, demonstrating a reduction in computation time while maintaining similar downstream behaviour to actual detectors.

In addition, PEMs have been extended to cooperative perception scenarios, studying how vehicle-to-everything communication can improve safety by compensating for perception limitations through information sharing between vehicles and infrastructure.

Adaptive simulation using perception error models

PEMs have been deployed in emergency braking scenarios using efficient importance sampling strategies to estimate rare collision probabilities. This enables likely and safety-critical perception errors to be identified.

In a similar fashion, a method has been described to identify perception errors that score highly on standard quality metrics but cause planning failures, termed adversarial perception errors. Using a boundary-attack algorithm on black-box planners in the CARLA simulator, such errors were shown to be systematically constructable, highlighting limitations of standard perception metrics for predicting downstream safety. This adversarial approach was extended with EMPERROR, a transformer-based generative PEM using the conditional variational autoencoder framework.