# [Bug/Feature Request] NAVSIM checkpoint incompatible with nuPlan/Waymo preprocessing pipeline

[nican2018]

Hello, thank you for this interesting work and for making it open source.

Issue Summary

I'm attempting to reproduce the NAVSIM evaluation results using the provided checkpoint (AutoVLA_PDMS_89.ckpt), but the preprocessing pipeline (tools/preprocessing/nocot_sample_generation.py/ cot_sample_generation.py) is incompatible with NAVSIM data format. There is a fundamental mismatch between the NAVSIM-trained checkpoint and the nuPlan/Waymo preprocessing code.

What I did

1. Downloaded the NAVSIM checkpoint: AutoVLA_PDMS_89.ckpt
2. Downloaded NAVSIM dataset with sensor_blobs and navsim_logs
3. Attempted to run evaluation using run_pdm_score_cot.py
4. Hit preprocessing failure due to missing NavSim dataset class

What I expected

• Preprocessing should support NAVSIM data format
• Evaluation should work with the provided NAVSIM checkpoint

After analyzing I realized that

There is a fundamental mismatch:

1. The checkpoint AutoVLA_PDMS_89.ckpt is trained/optimized for NAVSIM
2. The preprocessing code only supports nuPlan/Waymo data formats
3. No NavSim dataset class exists in the preprocessing module

My Questions

1. Is there a preprocessing script specifically for NAVSIM that I'm missing?
2. Should I use a different checkpoint for nuPlan/Waymo evaluation? Are those checkpoints available?
3. Is there documentation on how to adapt the preprocessing for NAVSIM?
4. What's the recommended way to reproduce the NAVSIM results (PDMS on NAVSIM mentioned in the paper)?

Environment

• AutoVLA version: latest from main branch
• NAVSIM dataset
• Hardware: RTX 5090, 32GB