FELT: Generating… | arXiv Physical AI Research Summary

1. Key Themes

Synthesizing Touch from Vision Alone

FELT's core achievement is generating physically consistent tactile pressure images from a single RGB frame, using a frozen DINOv2 visual encoder and a lightweight attention-based decoder. The model predicts per-finger contact probability and pressure intensity maps (12×32 grids per finger) in a single feed-forward pass with ~20ms latency on an RTX 4090. This means a robot can "feel" contact without a physical tactile sensor at inference time. As stated in Section 4.2: "At inference time, FELT needs only RGB data and can provide either generated tactile images or latent tactile features to downstream policies."

Sensor-Free Visuo-Tactile Policy Learning

The most commercially significant result is the "Vision + FELT Features" configuration, which uses NO real tactile sensor during either policy training or deployment. Instead, it uses intermediate latent features from the tactile generator as a compact contact representation. On Triangle Peg Insertion, this configuration achieved 90% final success vs. 70% for real tactile sensors and 50% for vision-only (Table 2). The authors hypothesize in Section 6.2: "With only 60 demonstrations per task, the variability and noise in real tactile images may limit their effectiveness as a stable learning signal for the policy. Features from G_ϕ, pretrained on large paired RGB-tactile data, may instead provide a smoother averaged contact representation that is easier to use in this low-data regime."

Physical Sensor Topology Awareness Matters

FELT explicitly models the dual-finger topology of tactile sensors through separate decoder branches for left and right panels, with a gated cross-panel exchange module. Ablations in Table 1 show that removing the dual-branch decoder causes Frame Accuracy to drop from 0.816 to 0.616 and Panel Asymmetry Error to triple from 0.738 to 2.225. The paper states in Section 6.1: "Removing the dual-branch decoder causes the model to severely underestimate tactile magnitude... confirming the importance of respecting the physical sensor topology."

Tactile Benefits Concentrate in Contact-Critical Stages

Across all four tasks, tactile information (real or generated) had minimal impact on grasping but large impact on reorientation, insertion, nesting, and wiping. For example, in Cup Nesting, vision-only achieved 55% success at Nest 2→3, while real tactile achieved 80% and FELT Features achieved 60% (Table 2). The paper notes in Section 6.2: "The per-stage breakdowns show that tactile information mainly improves contact-critical stages, with similar grasp success across methods but larger gains during reorientation, insertion, nesting, and wiping."

2. Contrarian Perspectives

Generated Tactile Features Can Outperform Real Tactile Sensors in Low-Data Regimes

Most robotics companies assume that real sensor data is always better than synthetic data. This paper challenges that assumption. On Triangle Peg Insertion, FELT's latent features (no real tactile sensor at any stage) achieved 90% success vs. 70% with real tactile sensors (Table 2). The authors argue that in low-data settings (60 demonstrations), real tactile sensor noise and variability can actually hurt policy learning, while pretrained latent features provide a cleaner signal. As stated in Section 6.2: "Features from G_ϕ, pretrained on large paired RGB-tactile data, may instead provide a smoother averaged contact representation that is easier to use in this low-data regime."

A Generic Visual Backbone Cannot Substitute for Tactile-Specific Training

One might assume that simply adding a second pretrained visual encoder (like DINOv2) as an auxiliary input would capture similar contact information. The paper directly tests this (Configuration 7, "Vision + DINOv2 Features") and finds it underperforms FELT Features on all four tasks and provides "no measurable benefit over Vision Only on Cup Nesting and Eraser Wiping" (Section S7.3). This means the tactile generator's training on paired RGB-tactile data produces contact-relevant representations that generic vision models cannot replicate, even when using the same backbone.

Zero Tactile Input Actively Degrades Performance Below Vision-Only

A policy trained with real tactile data but deployed with all-zero tactile input performs WORSE than a policy that was never trained with tactile data at all. On Tube Insertion, Zero Tactile achieved 20% vs. 40% for Vision Only (Table S7). The paper states in Section S7.3: "This indicates that the policy learns to rely on tactile input during training. Replacing it with uninformative zeros disrupts the learned representation and degrades performance below even the Vision Only baseline." This implies that visuo-tactile policies create hard dependencies on the tactile modality — you cannot simply remove the sensor at deployment without consequences, which makes FELT's ability to generate plausible tactile signals at inference time all the more valuable.

3. Companies Identified

Starpilot, Tactile sensor / robotics company, Provided research support and is acknowledged as a sponsor. The paper uses FlexiTac tactile sensors (developed by co-author Binghao Huang) which appear related to Starpilot's work. The paper states: "Daniel Seita acknowledges generous support from Starpilot, Google DeepMind, NVIDIA, Dexmate, Samsung Research America, and Honda Research Institute."

Google DeepMind, AI research lab, Provided research support. Relevant as a potential competitor or collaborator in visuo-tactile manipulation, given their work on foundation models for robotics.

NVIDIA, GPU / AI computing company, Provided research support and hardware (RTX 4090 used for inference). Relevant as the compute platform for real-time tactile generation at ~20ms latency.

Dexmate, Robotics company, Provided research support. Relevant as a potential deployment target for dexterous manipulation with visuo-tactile capabilities.

Samsung Research America, Electronics / robotics research, Provided research support. Relevant for potential integration of tactile sensing into consumer or industrial robotics platforms.

Honda Research Institute, Automotive / robotics research, Provided research support. Relevant for potential applications in manufacturing automation.

Analog Devices, Semiconductor company, Partially supported the work. Relevant as a potential supplier of tactile sensor electronics or signal processing components.

4. People Identified

Daniel Seita, University of Southern California (Viterbi School of Engineering), Senior author and corresponding contact. His lab focuses on visuo-tactile manipulation and data augmentation for robot learning. He has multiple recent papers on bimanual manipulation and data generation (CRAFT, D-CODA). Acknowledged support from Starpilot, Google DeepMind, NVIDIA, and Dexmate suggests strong industry connections.

Yunzhu Li, Columbia University, Co-author and developer of FlexiTac tactile sensors (referenced as [22]) and the 3D-ViTac system (referenced as [23]). His lab is a key player in open-source tactile sensing hardware and visuo-tactile learning. The FELT framework is built directly on his group's tactile sensor and dataset infrastructure.

Binghao Huang, Columbia University, Co-author and developer of FlexiTac sensors. His work on low-cost, open-source tactile sensing (referenced as [22]) is the hardware foundation for this paper. Relevant for anyone evaluating tactile sensor platforms.

Stefanos Nikolaidis, University of Southern California, Co-author focusing on human-robot interaction and robot learning. His lab contributes to the policy learning and evaluation methodology.

Zinan Li and Yiyang Ling, University of Southern California, Equal-contribution lead authors. They implemented the FELT architecture, collected the xArm datasets, and ran the real-robot experiments.

Yuming Gu, University of Southern California, Project lead for FELT. Listed as the corresponding project lead (marked with ‡ in the author list).

5. Operating Insights

You Can Deploy Visuo-Tactile Policies Without Tactile Sensors on the Robot

The most actionable finding for a CTO is that FELT enables three distinct deployment modes, and the most capital-efficient one (Vision + FELT Features) requires zero tactile hardware. The generator is trained once on existing paired visuo-tactile data (2,700 demonstrations from Zhu et al.), and then any vision-only robot can benefit from "imagined" tactile feedback. This means companies with existing vision-only datasets can retroactively add tactile-like capabilities without re-collecting data or instrumenting robots with fragile sensors. The paper confirms in Section 4.2: "This preserves spatial contact structure without requiring real tactile data during policy training or deployment."

Tactile Sensors Are a Data Collection Bottleneck, Not Just a Hardware Cost

The paper frames tactile data collection as the fundamental scaling problem for visuo-tactile manipulation. Section 1 states: "tactile data is particularly challenging to collect, as it requires specialized, fragile sensors and careful calibration." The generator was trained on 2.6M paired RGB-tactile frames from a handheld UMI-style device, but downstream policies only needed 60 demonstrations per task. This suggests a strategy: invest in a one-time high-quality paired visuo-tactile dataset to train a generator, then scale policy training using cheap vision-only data augmented with generated tactile signals.

Real-Time Tactile Generation Is Feasible at Control Rate

The generator runs at ~20ms on a single RTX 4090 (Section 6.2), and policies execute at 10 Hz. This means FELT can operate in the control loop without introducing unacceptable latency. For companies building contact-rich manipulation systems, this validates that cross-modal tactile generation is not just an offline data augmentation tool but can serve as a real-time sensor replacement during deployment.

6. Overlooked Insights

FELT Requires the Contact Region to Be Visible in RGB — Heavy Occlusion Breaks It

The paper acknowledges in Section 7: "our method relies on visible contact regions in RGB observations. Heavy occlusion can make FELT fail to infer tactile information." This is a significant deployment limitation: in many real-world manipulation scenarios (e.g., deep bin picking, in-hand manipulation with occluding fingers), the contact region is precisely where vision fails. The paper suggests multi-camera setups or temporal models as solutions, but neither is implemented. For investors evaluating this technology, the key question is whether the target manipulation scenarios maintain visual line-of-sight to the contact zone.

Temporal Instability of Generated Tactile Signals Is an Unaddressed Problem

FELT predicts each frame independently, which can produce jumpy tactile signals even when true contact varies smoothly. Section S9 states: "Such discontinuities may degrade downstream policy performance, particularly during sustained contact phases such as wiping or insertion." This is particularly relevant for tasks like Eraser Wiping, where sustained, stable contact pressure is essential. The paper does not implement temporal smoothing or history conditioning, leaving this as an open problem that could limit deployment in precision contact tasks.