CoRA-VTLA: Contact-Guided Attention and Queryable Response Memory for Tactile Post-Training

Chengye Song 1
Rui Song 1
Shengqi Wang 1
Jiawei Zhang 2
Hongzhou Wang 2,4
Xueyang Guo 3
Yang Liu 1
Preprint
1Dalian University of Technology, 2Jilin University, 3Tsinghua University, 4Enomind Technology

Abstract

In precision manipulation, small errors in alignment or force can cause jamming or deformation, making contact feedback essential for adapting pretrained vision–language–action (VLA) models to these tasks. A single tactile observation leaves the response to recent actions ambiguous, and directly introducing tactile inputs during post-training can disrupt general-purpose capabilities acquired during pretraining and draw visual attention away from contact regions. To address these challenges, CoRA-VTLA augments pretrained VLAs with lightweight tactile modules. A queryable action–response memory retrieves recent contact responses so that the action expert can combine them with immediate tactile feedback. Contact-guided gaze supervision concentrates the visual attention of the tactile semantic token on image regions corresponding to fingertip contact. CoRA-VTLA achieves 73.33% mean success across three MuJoCo tasks, compared with 50% for direct tactile fusion with the same sensor inputs. It also achieves 67% success on an optical-tactile task from the public UniVTAC benchmark and 66.7% mean success across three real-robot tasks.

CoRA-VTLA at a glance

Overview: blocked insertions, the three tactile pathways, success rates, and rollout keyframes.

(a) Blocked insertions with fingertip pressure maps and load traces around withdrawal. (b) A tactile semantic token enters the VLM, while current contact and queried action–response memory condition the action expert. (c) Success rates in MuJoCo, on UniVTAC Insert Tube, and on three real-robot tasks. (d) Rollout keyframes.

Method

CoRA-VTLA architecture with the semantic branch, contact token, interaction memory token, and response pretraining.

CoRA-VTLA adds three lightweight tactile pathways to a pretrained π0.5 policy: a current contact token carrying the instantaneous pressure and wrist force/torque, a queryable action–response memory token that encodes the last 0.8 s of measured contact responses and is read with a query built from the current contact and robot state, and a tactile semantic token appended to the VLM prefix, grounded by contact-guided gaze supervision.

The two action-side encoders are first initialized by response pretraining: from task demonstrations alone, they are trained to predict the next-step change in pressure, wrist force/torque, and robot motion, after which the temporary prediction head is discarded. All modules are then trained jointly with the full policy under the standard flow-matching objective, with attribute and gaze supervision on the semantic branch.

Simulated rollouts

Plug insertion

StartFirst contactAdvancingInsertedReleasedDone

A plug insertion completed by CoRA-VTLA.

Knob operation

StartGraspPress and turnTarget angleReleasedDone

CoRA-VTLA grasps the knob, presses to unlock, and turns it to the target angle.

Nail insertion

StartFirst contactAdvancingInsertedReleasedDone

A nail insertion completed by CoRA-VTLA.

The signals behind a recovery

Fingertip load over timeWrist wrench over timeEnd-effector displacement over time

A successful plug insertion: fingertip load, wrist wrench, and realized displacement — the responses the memory token reads.

Rollouts of CoRA-VTLA

Recovery after blocked contact on plug, knob, and nail insertion.

Simulated behavior and attention analysis.
Simulated recovery behavior with synchronized contact signals and attention.

Real robot

On the real robot, the vision-only baseline keeps pressing after contact is blocked, tilts or drops grasped objects, and trips force limits — while CoRA-VTLA registers the load response, adjusts, and completes the task.

Vision-only baseline (failures)

Pressing through the blockage until failure · force-limit trip · tilting the cup during stacking.

CoRA-VTLA (successes)

Withdrawing and realigning after a blockage · gentle pressing on contact · a complete task episode.

Platform

Gripper close-up with tactile arraysWorkspacePlatform viewPlatform viewPlatform view
Real-robot platform and rollouts for cup stacking, stamping, and tube insertion.

Real-robot tasks and rollouts. CoRA-VTLA achieves the highest success rate on all three tasks — cup stacking, stamping, and tube insertion — averaging 66.7%, 22.0 points above the strongest baseline.

Results

MethodMuJoCo (%)UniVTAC (%)Real robot (%)
NailPlugKnobAvg.Insert TubeCupStampTubeAvg.
Base (π0.5)74.00.020.031.334638303032.7
Direct68.032.050.050.005150483444.0
FTP-π0.556.04.072.044.005746444444.7
CoRA-VTLA78.076.066.073.336768765666.7

Success rates under identical demonstrations, evaluation scenes, and tactile inputs within each benchmark; optical tactile on UniVTAC.

Where the policy looks

Contact-region attention with and without gaze supervision on matched observations.

Contact-region attention on matched observations. With gaze supervision, the tactile semantic token places 52.38% of its main-camera attention inside the projected fingertip contact region at the supervised layer; without gaze, 2.77%.

UniVTAC Insert Tube results.
UniVTAC Insert Tube: paired outcomes and per-episode comparison.

CoRA-VTLA on UniVTAC Insert Tube with optical tactile sensing.

Citation

@misc{song2026coravtla,
author = "{Chengye Song and Rui Song and Shengqi Wang and Jiawei Zhang and Hongzhou Wang and Xueyang Guo and Yang Liu}",
title = "CoRA-VTLA: Contact-Guided Attention and Queryable Response Memory for Tactile Post-Training",
year = "2026",
howpublished = "\url{https://arxiv.org/abs/2611.23963}",
}