More Works from Our Lab

WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Users

Synthetic data to improve pose estimation for people in wheelchairs.

ACM CHI 2024
Accessibility Scout: Personalized Accessibility Scans of Built Environments

Agentic reasoning and workflows for personalized accessibility assessments.

ACM UIST 2025

DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

William Huang¹, Siyou Pei¹, Leyi Zou¹, Eric J. Gonzalez², Ishan Chatterjee², Yang Zhang¹
¹University of California, Los Angeles
²Google
ACM CHI 2026
arXiv Code Model Weights Dataset Sample

Abstract

The proliferation of XR devices has made egocentric hand pose estimation a vital task, yet this perspective is inherently challenged by frequent finger occlusions. To address this, we propose a novel approach that leverages the rich information in dorsal hand skin deformation, unlocked by recent advances in dense visual featurizers. We introduce a dual-stream delta encoder that learns pose by contrasting features from a dynamic hand with a baseline relaxed position. Our evaluation demonstrates that, using only cropped dorsal images, our method reduces the Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers >=50% occluded) compared to state-of-the-art techniques that depend on the whole hand's geometry and large model backbones. Consequently, our method not only enhances the reliability of downstream tasks like index finger pinch and tap estimation in occluded scenarios but also unlocks new interaction paradigms, such as detecting isometric force for a surface "click" without visible movement while minimizing model size.

Overview figure illustrating DeltaDorsal and dorsal hand feature-based pose estimation
DeltaDorsal uses dense visual featurizers to extract dorsal skin deformations from the back of the hand. By comparing the visual features from a neutral and current hand pose, DeltaDorsal isolates these changes and then uses this to predict hand posture.

BibTeX


        @misc{huangDeltaDorsalEnhancingHand2026,
        title = {{{DeltaDorsal}}: {{Enhancing Hand Pose Estimation}} with {{Dorsal Features}} in {{Egocentric Views}}},
        shorttitle = {{{DeltaDorsal}}},
        author = {Huang, William and Pei, Siyou and Zou, Leyi and Gonzalez, Eric J. and Chatterjee, Ishan and Zhang, Yang},
        year = 2026,
        month = jan,
        number = {arXiv:2601.15516},
        eprint = {2601.15516},
        primaryclass = {cs},
        publisher = {arXiv},
        doi = {10.48550/arXiv.2601.15516},
        urldate = {2026-02-25},
        archiveprefix = {arXiv}
      }