TacGraph

Simultaneous Extrinsic Contact and In-Hand Pose Estimation via Distributed Tactile Sensing

Mark Van der Merwe1, Kei Ota2, Dmitry Berenson1, Nima Fazeli1, Devesh Jha3
1University of Michigan, 2Mitsubishi Electric 3Mitsubishi Electric Research Laboratories
IEEE Robotics and Automation Letters 2026
arXiv

TacGraph utilizes visual and distributed tactile feedback to estimate in-hand object pose and extrinsic contact. We utilize factor graphs to enforce physical constraints and to efficiently solve.

Abstract

Prehensile autonomous manipulation, such as peg insertion, tool use, or assembly, require precise in-hand understanding of the object pose and the extrinsic contacts made during interactions. Providing accurate estimation of pose and contacts is challenging. Tactile sensors can provide local geometry at the sensor and force information about the grasp, but the locality of sensing means resolving poses and contacts from tactile alone is often an ill-posed problem, as multiple configurations can be consistent with the observations. Adding visual feedback can help resolve ambiguities, but can suffer from noise and occlusions.

In this work, we propose a method that pairs local observations from sensing with the physical constraints of contact. We propose a set of factors that ensure local consistency with tactile observations as well as enforcing physical plausibility, namely, that the estimated pose and contacts must respect the kinematic and force constraints of quasi-static rigid body interactions. We formalize our problem as a factor graph, allowing for efficient estimation. In our experiments, we demonstrate that our method outperforms existing geometric and contact-informed estimation pipelines, especially when only tactile information is available.

Method Overview

TacGraph method overview

We propose TacGraph - a method for estimating object pose and extrinsic contacts during prehensile manipulation using distributed tactile feedback and (optionally) visual feedback. Our method has two main components. First, we have a series of learned tactile modules. These translate raw tactile signals to useful intermediates (geometry, forces, and displacements). Second, we have a Factor Graph which takes the learned tactile outputs, along with known object/environment geometries and enforce the physical constraints of contact (geometric consistency, non-penetration, contact kinematics, and force balance).

Results - Tactile Only

We show comparison of predicted and ground truth pose and predicted and ground truth extrinsic contact estimates from tactile only feedback. The results indicate that the contacts are utilized by our proposed framework to narrow down and disambiguate the local tactile feedback, yielding highly accurate predictions.

Application: Open-Loop Insertion

(a) TacGraph (Ours)
(b) ICP
(c) CHSEL
(d) SCOPE (v2)

We demonstrate application of TacGraph for open-loop insertion with tactile-only feedback. We perform a series of exploratory pokes and run inference with each method, then use the final estimate to perform an open loop insertion with low tolerance (~3mm). We show comparison to several baselines. Our method is the only to succeed on this example and has the highest success rate across 4 test objects (see paper for details). We show the final estimate of each method against the ground truth and the attempted insertion.

BibTeX

@article{vdm2026tacgraph,
  author    = {Van der Merwe, Mark and Ota, Kei and Berenson, Dmitry and Fazeli, Nima and Jha, Devesh},
  title     = {Simultaneous Extrinsic Contact and In-Hand Pose Estimation via Distributed Tactile Sensing},
  journal   = {IEEE Robotics and Automation Letters},
  year      = {2026},
}