OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher

arXiv preprint, 2026

Damiano Da Col^1,2, Maximilian Igl³, Peter Karkus³, Kashyap Chitta^1,4*, Boris Ivanovic³, Marco Pavone^3,5, Konrad Schindler², Christos Sakaridis²

¹KE:SAI ²ETH Zürich ³NVIDIA Research ⁴ELLIS Institute Tübingen ⁵Stanford University
^*Work done in part while at NVIDIA Research

Paper (arXiv) Code (coming soon)

OPTED overview: rendered sensor input seen by the student on top, HD map and agents seen by the teacher below — A pre-trained end-to-end student drives closed-loop in a sensor-based simulator (top). At every visited state, a privileged teacher trained with RL on vectorized inputs in a render-free simulator (bottom) provides the distillation target.

Abstract

As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6× and 9.5×, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior.

Method

The OPTED pipeline: teacher and student pre-training, then iterated closed-loop data collection, teacher labeling and student update — **The OPTED pipeline.** A privileged teacher is trained with RL in a render-free simulator on vectorized states. The end-to-end student is pre-trained with behavior cloning on human driving logs. Fine-tuning then iterates three steps: (1) the student drives the rendered simulator in closed loop; (2) each visited state is synchronized into the vectorized simulator, where the frozen teacher produces the target trajectory; (3) the student is updated on a FIFO buffer of recent rollouts with an imitation loss matched to its output head.

Results

On the 441 held-out NuRec scenes in AlpaSim, OPTED raises the scene score of LTFv6 from 26.5 to 41.8 and of VaVAM from 3.9 to 37.1, outperforming open-loop BC fine-tuning and closed-loop supervised fine-tuning with log-anchored targets at equal training budget. Both improvements transfer to the private AlpaSim challenge leaderboard. In a controlled study with vectorized students on WOMD, OPTED reaches 95% of the teacher's scene score in about 6.8k closed-loop episodes, against 6.2M for RL fine-tuning of the same student, while staying closer to the human jerk distribution.

Code

Code will be released at github.com/01Dami23/opted.

Citation

@article{dacol2026opted,
  title   = {{OPTED}: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher},
  author  = {Da Col, Damiano and Igl, Maximilian and Karkus, Peter and Chitta, Kashyap and Ivanovic, Boris and Pavone, Marco and Schindler, Konrad and Sakaridis, Christos},
  journal = {arXiv preprint arXiv:XXXX.XXXXX},
  year    = {2026}
}