RCT: A Robot-Collected Touch–Vision–Language Dataset for Tactile Generalization

Project page · Paper (arXiv:2606.31694) · Dataset (Figshare)

RCT (Robotic Contact Tactile) is a robot-collected touch–vision–language dataset: 29,279 DIGIT tactile frames in 1,832 contact sequences from full robot presses on 122 industrial reference materials, with per-frame contact force, one 2048×1536 photograph per material, and material-level tactile descriptors. The dataset is organized around contact sequences to enable controlled held-out evaluation across materials, categories, sensors, contact positions, and presses.

What's in this repository

Path	Contents
docs/	The project website (GitHub Pages), including a live demo scrubbing real contact sequences
tvl_modifications.patch	Our core method changes on top of the official TVL codebase (+190/−19 lines across 5 files: RCTDataset, multi-positive tactile→vision evaluation, chunked inference)
tools/splits/	Split-generation toolkit: material / category / axis (position, sensor) / trajectory hold-outs, density filters (full / uniform5 / deep5), caption variants
tools/leakage/	Leakage audits: contact-sequence overlap in the released TVL/HCT split, and the training-free raw-pixel nearest-neighbor audit
tools/analysis/	Density × leakage analysis, depth-signal diagnostic, depth-progression visualization
tools/results/	SLURM-log parsing → results.csv → paper-ready tables
experiments/	Declarative experiment spec (experiments.yaml + generator) and one representative SLURM script per run type
CHANGES.md	Complete map of every modification and new file, with per-file descriptions

Setup

Our code is an overlay on the official TVL codebase at a pinned commit:

git clone https://github.com/Max-Fu/tvl.git
cd tvl && git checkout f489503
git apply /path/to/RCT/tvl_modifications.patch
cp /path/to/RCT/tools/*/*.py tvl_enc/tools/

Then download the RCT dataset from Figshare and follow TVL's environment setup. Cluster-specific paths in experiments/ scripts are placeholders (/path/to/workspace) — point them at your own checkout, data, and output directories, or regenerate all run scripts from experiments/experiments.yaml with experiments/gen_experiments.py.

Reproducing the paper

• Splits — tools/splits/ generates every held-out setting (material K=2/5/20, category, position/sensor axis, trajectory-disjoint, frame-random control); density variants via make_density_filter.py (full / uniform5 / deep5).
• Leakage audits — tools/leakage/audit_tvl_trajleakage.py (sequence overlap in the released TVL/HCT split) and audit_hct_raw_pixel.py (training-free raw-pixel nearest-neighbor sequence recovery).
• Training / evaluation — runs are specified in experiments/experiments.yaml; experiments/examples/ holds one representative script per run type.
• Tables — tools/results/parse_eval_logs.py → make_paper_tables.py.

See CHANGES.md for the full file-by-file map.

Citation

@misc{he2026rct,
  title         = {{RCT}: A Robot-Collected Touch--Vision--Language
                   Dataset for Tactile Generalization},
  author        = {Jingbo He and Michael F{\"a}rber and Roberto Calandra},
  year          = {2026},
  eprint        = {2606.31694},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  doi           = {10.48550/arXiv.2606.31694},
  url           = {https://arxiv.org/abs/2606.31694},
}

License

• Code (this repository): Apache-2.0 — matching the upstream TVL codebase our patch builds on.
• Dataset (Figshare): CC BY 4.0.