Abstract
High infraction rates remain a major bottleneck for end-to-end (E2E) autonomous driving, as evidenced by the persistent gap between strong open-loop performance and limited closed-loop driving performance on the CARLA Leaderboard. Although collision-related infractions account for the majority of closed-loop failures, explicit collision-aware representation learning has received little attention. We propose VLAAD (Video-Language-Augmented Anomaly Detector), a lightweight collision-aware plug-in designed to augment pretrained E2E driving agents with explicit representations of collision risk. VLAAD learns these representations from captioned driving videos using multiple-instance learning (MIL), producing temporally localized collision-risk signals suitable for closed-loop policy conditioning. Learning such representations can benefit from multimodal supervision, yet existing simulator datasets lack both diverse collision scenarios and rich language annotations. We therefore introduce CARLA-Collide, a large-scale multimodal dataset of realistic collision and infraction events collected from closed-loop TF++ rollouts across diverse CARLA Leaderboard scenarios. VLAAD is designed for integration with pretrained E2E driving agents that expose analogous state representations. We demonstrate this plug-in integration on a pretrained TransFuser++ agent, requiring only minimal architectural modification and fine-tuning and achieving a 22.13% relative improvement in driving score on the CARLA Leaderboard. To evaluate real-world generalization, we further introduce Real-Collide, a multimodal benchmark of diverse dashcam videos paired with semantically rich annotations for collision-risk understanding. Using dense frame-level annotations on CARLA-Collide, we introduce a frame-level evaluation protocol for collision localization in ego-view autonomous-driving videos, enabling quantitative and qualitative assessment of temporal localization quality and demonstrating the benefits of MIL. Despite containing only 0.6B parameters, VLAAD outperforms the multi-billion-parameter vision-language model LLaVA-Next by 24.9 percentage points in AUC on the Real-Collide benchmark, even when the latter is fine-tuned on real-world driving data.