Our latest research, published in IEEE Access, introduces Li-ViP3D++, a framework that jointly detects and tracks road users and predicts their future trajectories. Developed by researchers from FIIT STU, Karlsruhe Institute of Technology and Halmstad University, the approach adaptively combines camera and LiDAR information to improve prediction accuracy while substantially reducing false detections.
Addressing Error Propagation in Autonomous Driving
Autonomous vehicles need to understand both where surrounding road users are and how they might move. Traditional systems divide this task into separate detection, tracking and prediction stages. Errors made early in the pipeline can propagate, while limited information exchange between modules can discard useful visual and geometric cues.
End-to-end models address these limitations by learning the tasks together. However, effectively combining cameras and LiDAR remains a challenge. Cameras provide rich visual context, while LiDAR supplies precise geometric information. Fusion methods based on fixed rules or discrete feature selection can restrict how effectively the model learns to use these complementary inputs.
What Is New?
Li-ViP3D++ introduces Query-Gated Deformable Fusion (QGDF), a mechanism that adapts sensor fusion to each individual road user.
The model maintains learned representations, known as agent queries, that connect perception, tracking and forecasting. For each query, QGDF gathers relevant visual information from multiple cameras, samples LiDAR features at learned locations and determines how strongly each modality should contribute.
This replaces heuristic fusion and discrete selection steps with a fully differentiable process that can be optimised jointly with the rest of the network. Rather than assigning the same sensor weighting to every object, the model learns to balance visual and geometric evidence for each agent.
More Accurate Predictions, Fewer False Detections
The team evaluated Li-ViP3D++ on the nuScenes benchmark, predicting six possible future trajectories per agent over a six-second horizon. Experiments covered seven categories, including cars, trucks, buses, motorcycles, bicycles and pedestrians.
Compared with the preceding Li-ViP3D model under the original input configuration, Li-ViP3D++ achieved:
- End-to-end Prediction Accuracy (EPA) of 0.505, up from 0.250.
- A false-positive ratio of 0.069, down from 0.221—a reduction of approximately 69%.
- Minimum average trajectory displacement error of 1.11 metres, improved from 1.45 metres.
- Minimum final displacement error of 1.80 metres, improved from 2.20 metres.
The model also achieved a detection mAP of 0.616. These results matter because reliable forecasting depends on identifying the correct road users as well as estimating their future motion. False detections can otherwise introduce unnecessary obstacles into downstream planning.
The improvements came with a modest reduction in inference time: 139.82 ms versus 145.91 ms for Li-ViP3D on the reported hardware.
Strong Results with Lower-Resolution Images and Without HD Maps
Additional experiments tested reduced camera resolution, removal of HD map input and both constraints together. Li-ViP3D++ retained its advantages over the evaluated ViP3D and Li-ViP3D baselines across these settings.
With lower-resolution images and no HD map input, inference time fell to 75.40 ms, measured on an NVIDIA RTX 4090 GPU. This configuration illustrates a useful trade-off between input requirements, computation and prediction quality.
The findings support further investigation of deployments where detailed maps or high-resolution imagery are difficult to maintain. Evaluation under adverse weather and lighting conditions remains future work.
Publication in IEEE Access
The paper was published on 1 July 2026 in IEEE Access, IEEE’s multidisciplinary, peer-reviewed, fully open-access journal. Its publication criteria emphasise originality, technical correctness and clear presentation, making it an established venue for applied engineering and computing research.
IEEE’s official bibliometrics page reports a Journal Impact Factor of 4.2 and a 2025 CiteScore of 9.3. These indicators describe the journal’s citation visibility; the contribution of this particular work is supported by its benchmark comparisons, component ablation studies and publicly available source code.
Li-ViP3D++ advances our research into multimodal scene understanding by showing how adaptive fusion can improve both the detection of road users and the prediction of their movements.
Authors: Matej Halinkovic, Nina Masarykova, Alexey Vinel and Marek Galinski.
Read the paper: Li-ViP3D++: Query-Gated Deformable Camera–LiDAR Fusion for End-to-End Perception and Trajectory Prediction
Source code: GitHub