MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving

This is the code for paper "MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving" (T-TITS 2025).

MSSF is a simple but effective multi-stage sampling fusion network based on 4D radar and camera for 3D object detection.

🔧 Install

You can prepare environment via conda:

conda env create -f environment.yml -n mssf

Clone our repository and install (based on MMDetection3D):

pip install -e . -v

📚 Dataset Preparation

1. Download Datasets

We use VoD and TJ4DRadSet in our experiments. Please download the dataset and put them in data. We organize data as follows:

data
 |
 |-- vod
 |     |—- ImageSets
 |     |-- testing
 |     |-- training
 |             |
 |             |-- calib
 |             |-- image_2
 |             |-- label_2
 |             |-- pose
 |             |-- velodyne
 |--- tj4d
 |     |—— ImageSets
 |     |-- testing
 |     |-- training
 |              |-- calib
 |              |-- image_2
 |              |-- label_2
 |              |-- velodyne

2. Generate pickle info and reduced points (in FoV)

First, we should generate pickle info and reduced points in FoV, which is common process in mmdetecton3d and OpenPCDet codebases.

VoD dataset: python tools/create_data.py --dataset vod --root-path ./data/vod --out-dir ./data/vod
TJ4D dataset: python tools/create_data.py --dataset tj4d --root-path ./data/tj4d --out-dir ./data/tj4d

⚠️ Absolute radial velocity for TJ4DRadSet

The original TJ4DRadSet provides radar points with an 8-channel feature dimension. Following RCFusion and RadarMFNet, we compute and append the absolute radial velocity, expanding the channel dimension to 9. The computed absolute radial velocity can be downloaded from 🤗HF. To read it: np.fromfile('000000.bin', dtype=np.float32)

🚀 Run

Checkpoint

Please download checkpoints & configs, and put them in exps.

Evaluation

To run evaluation:

python tools/test.py $CONFIG --checkpoint $CHECKPOINT --metric [VoDMetric,TJ4DMetric]

VoD Dataset

Method	mAP @ EAA	mAP @ DC
MSSF-V	59.96	81.32
MSSF-PP	63.31	79.78

TJ4DRadSet Dataset

Method	mAP 3D	mAP BEV
MSSF-V	37.97	43.11
MSSF-PP	41.75	48.41

Training

python tools/train.py $CONFIG --work-dir $WORK_DIR

🙏🏻 Acknowledgement

MMDetection3D, OpenPCDet, VoD dataset, TJ4DRadSet dataset

⭐️ Citation

If you find our paper and code useful for your research, please consider citing us and giving a star to our repository:

@ARTICLE{10947638,
  author={Liu, Hongsi and Liu, Jun and Jiang, Guangfeng and Jin, Xin},
  journal={IEEE Transactions on Intelligent Transportation Systems}, 
  title={MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving}, 
  year={2025},
  volume={26},
  number={6},
  pages={8641-8656},
  doi={10.1109/TITS.2025.3554313}}