ROSBENCH: A New Benchmark for Robust Autonomous Vehicle Perception

By Automotive Innovation Lab

Published at the IEEE-indexed ELMAR 2025 Conference (Zadar, Croatia)
Authors: Matej Halinkovic, Miroslav Kunovsky, Marek Galinski
Institution: Slovak University of Technology, Bratislava

What We Did

Our team presented ROSBENCH, a simulation-based benchmark designed to test the robustness of autonomous vehicle (AV) perception systems under diverse sensor qualities and environmental conditions. Built on top of the CARLA simulator, ROSBENCH allows researchers to systematically vary parameters such as sensor resolution, LiDAR beam count, weather, lighting, and noise — without resorting to artificial post-processing that can distort results.

The benchmark replicates the structure of the popular nuScenes dataset, ensuring full compatibility with existing AV perception models and evaluation tools.

Why This Dataset Matters

Real-world AV datasets like KITTI, Waymo, and nuScenes have driven major progress in computer vision and sensor fusion. However, they all share two limitations:

  1. Fixed sensor quality – they don’t capture how perception systems behave with lower-resolution or noisy sensors.
  2. Limited environmental variability – conditions such as night driving, heavy rain, or fog are underrepresented.

ROSBENCH fills this gap by introducing a configurable, reproducible, and artifact-free simulation benchmark, making it possible to:

  • Evaluate how perception and prediction models perform under realistic degradations.
  • Study the trade-offs between sensor cost and perception performance.
  • Support sim-to-real transfer research, helping models trained in simulation generalize better to the real world.

The dataset includes:

  • 500 scenes across multiple CARLA maps, five weather presets, and varying lighting conditions.
  • Three sensor quality levels for both LiDAR and cameras (High, Mid, Low).
  • 1.8 million images, 300,000 LiDAR point clouds, and 1.3 million annotated bounding boxes across seven object categories.

Key Outcomes

  • First benchmark to combine sensor quality variation, 360° RGB and LiDAR coverage, and rich environmental diversity.
  • Compatible with NuScenes format, enabling seamless integration into existing workflows.
  • Supports full Perception and Prediction pipelines, including object detection, tracking, and trajectory forecasting.
  • Publicly available code and dataset, promoting transparency and open benchmarking in AV research.

Why It’s Valuable for the Community

ROSBENCH provides a controlled, flexible, and open environment for testing AV perception systems. It enables the community to:

  • Benchmark model robustness against real-world variability.
  • Determine minimum viable sensor configurations for safe operation.
  • Advance simulation-based validation, a critical step toward reliable autonomous driving.

By bridging the gap between pristine simulation data and unpredictable real-world conditions, ROSBENCH moves AV perception research closer to practical deployment.

Acknowledgment

This research was supported by the EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia (Project No. 09I05-03-V02-00014) and by the APVV-23-0519 project “Legal and Technical Challenges of Smart Mobility to Increase Road Traffic Safety,” and by the internal grant scheme of the Slovak University of Technology to support Young Researchers.