DriveReferee

Abstract

Generative world-action models (WAMs) jointly generate future video and vehicle actions, while their action branches remain primarily optimized by expert imitation. Yet imitation provides no explicit closed-loop geometric verdict for generated trajectories, making verification important during both training and deployment. Closed-loop evaluators can check collision and drivable-area violations, but require privileged scene state unavailable at deployment. Existing approaches often address this gap by learning a verifier from sensor features. For these geometric checks, the rule itself is explicit. For example, collision is determined by whether the rolled-out ego footprint overlaps occupied vehicle space. What is unavailable at deployment is the scene state needed to apply the rule.

We introduce DriveReferee, which uses a learned geometry readout to predict the scene representation from camera observations and executes the geometric safety rule directly rather than learning it. The resulting analytic referee evaluates collision and drivable-area safety from a scene state and candidate trajectory. During training, it scores self-sampled trajectories on ground-truth state and distills the resulting preferences into the WAM policy. At deployment, the same referee evaluates generated trajectories on this predicted state and selects a safer alternative when needed. The analytic referee requires no verdict-specific training, and its decisions follow an explicit geometric rule. Under matched candidates and inference budgets, it matches or outperforms all learned-verifier and heuristic baselines. Given the same predicted state and trajectory, learning the verdict provides no measurable downstream gain despite requiring tens of thousands of evaluator-labeled training examples. On the full NAVSIM navtest, DriveReferee reaches 92.02 PDMS with single-camera visual input and no external training data.

Method

DriveReferee overview: policy, perception readout, and analytic referee.

The policy generates future video and candidate plans; the readout predicts the BEV scene state (drivable area and time-indexed vehicle occupancy) from frozen visual features and ego-pose history. The analytic referee, with no learned parameters, is placed twice: at training on ground-truth maps to form preference pairs that are distilled into the generator, and at deployment on the predicted map to gate the default plan and, on alarm, rank further samples on the same map.

World-action policy. A frozen reasoner tower plus a trained generation tower produce a future video and a decoded plan per sample, so repeated sampling yields candidate trajectories without a separate proposal network.
Perception readout. A lightweight head on frozen visual features predicts the drivable-area and vehicle-occupancy maps that the referee needs, using privileged rasterized annotations as training supervision only.
Analytic referee. Kinematic rollout with the evaluator's LQR tracker and bicycle model, footprint checks at four corners and the center, and metric clearance margins from a distance transform. Zero learned parameters.

Generated Future Videos

Future front-camera videos generated by the world–action policy on NAVSIM navtest scenes, shown next to the recorded ground truth. Each clip covers the next 4 seconds at 2 Hz from 832×480 observations.

Results

Comparison with representative world-model-based planners on NAVSIM navtest (12,146 scenes). N×C denotes N cameras and +L denotes LiDAR. PDMS follows NAVSIM v1 and EPDMS follows v2. With single-camera visual input and no external training data, the complete system reaches 92.02 PDMS and 91.56 EPDMS, outperforming the listed generative world-action models on both reported official metrics.

Method	Input	Video	NC↑	DAC↑	TTC↑	Comf.↑	EP↑	PDMS↑	EPDMS↑
DrivingGPT	1×C	✓	98.9	90.7	94.9	95.6	79.7	82.4	—
WoTE	3×C+L	✗	98.5	96.8	94.9	99.9	81.9	88.3	—
DriveVLA-W0	1×C	✓^†	98.7	99.1	95.3	99.3	83.3	90.2	86.1
PWM	1×C	✓	98.6	95.9	95.4	100.0	81.8	88.1	—
DriveLaW^‡	1×C	✓^†	99.0	97.1	96.7	100.0	81.3	89.1	—
CoPhy^§	C+L	✗	99.0	98.2	96.8	100.0	85.3	91.4	86.1
DriveDreamer-Policy	3×C	✓	98.4	97.1	95.1	100.0	83.5	89.2	88.7
Metis	1×C	✓	98.3	97.1	94.7	100.0	83.4	89.1	89.5
DriveVA^‡	1×C	✓	99.2	97.5	98.7	100.0	83.5	90.9	—
UNIVERSE	1×C	✓^†	99.1	97.6	98.5	100.0	83.6	91.0	—
Ours (base, 4B backbone)	1×C	✓	99.22	97.37	96.86	100.00	84.09	90.47	90.03
Ours (full, 4B backbone)	1×C	✓	99.21	98.25	96.86	100.00	85.36	91.50	91.08
Ours (base, 16B backbone)	1×C	✓	99.31	97.79	97.41	100.00	84.45	91.08	90.69
Ours (full, 16B backbone)	1×C	✓	99.49	98.64	97.91	100.00	84.99	92.02	91.56

^†Trained with video generation but no video is generated at test time. ^‡DriveVA uses extra CARLA data; DriveLaW uses nuPlan and nuScenes. ^§CoPhy uses external VQA data and multi-candidate inference. Metis reports single-pass inference; its oracle best-of-6 result is excluded. "Full" denotes referee distillation with K=2 gated selection. Our 4B/16B models use Cosmos3-Edge/Cosmos3-Nano, with 2B/8B trainable generation towers; the reasoner towers are frozen.

Does the verdict need to be learned?

Matched-budget comparison of verdict sources.

Per-scene paired ΔEPDMS relative to the imitation-only base on navtest. All sources use the same pre-sampled K=2 candidates and are calibrated to the same average sampling budget. Error bars are bootstrap 95% confidence intervals.

Verdict-supervision scaling in the same-map setting.

Same-map setting, varying only the amount of verdict-training data; 100% is 66,385 training candidates. The analytic referee uses no verdict-training labels.

Under matched candidates and a matched sampling budget, the best learned verifiers improve EPDMS by +0.23 and +0.26, while the analytic referee reaches +0.30. None of the learned verifiers performs significantly better.
In the same-map control, where a learned verifier receives the same predicted map, candidate trajectory, and motion state as the analytic referee, 66,385 evaluator-labeled training candidates give a paired difference of −0.01 ΔEPDMS (95% CI [−0.15, +0.14]), and the two select the same plan in 83.4% of scenes.
The same referee in two placements: training-time preference distillation gives +0.92 ΔEPDMS (95% CI [+0.71, +1.13]) over the imitation-only base, and deployment-time gated selection on the undistilled base gives +0.30 ([+0.18, +0.42]).
Referee distillation preserves video-generation quality: FVD 23.90 → 24.25, LPIPS 0.4166 → 0.4126, PSNR 18.75 → 18.92 on the same scenes, protocol, and seeds.

Deployment-Time Verification

Verification on self-predicted maps with the base policy and K=2. (a) One intervention example: at the flagged state the default plan crosses the predicted drivable boundary by 0.40 m, while the referee-selected plan keeps 1.26 m clearance. The signed-distance curve and the projected boundaries show the same failure in BEV and in video space. (b) Four additional replacements found by automatic screening, where the default plan leaves the ground-truth drivable area while the selected plan stays inside both the predicted and ground-truth boundaries.

BibTeX

@article{yu2026drivereferee,
  title   = {DriveReferee: Geometric Safety Verdicts Need Not Be Learned
             for Driving World-Action Models},
  author  = {Yu, Fengcheng and Parikh, Dhruv and Ye, Junjie
             and Bhatt, Maulik and Vu, Thang and Vasiljevic, Igor
             and Guizilini, Vitor and Wang, Yue},
  journal = {arXiv preprint arXiv:2609.22762},
  year    = {2026}
}

DriveReferee: Geometric Safety Verdicts Need Not Be Learned for Driving World-Action Models