HandMvNet

VISAPP 2025

HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion

Muhammad Asad Ali Nadia Robertini Didier Stricker

RPTU Kaiserslautern-Landau · German Research Center for Artificial Intelligence (DFKI)

HandMvNet reconstructs absolute 3D hand joints and meshes from multiple RGB views using joint-aligned feature sampling and cross-attention fusion. The model runs in real time, handles strong hand-object occlusion, and can operate without camera parameters.

Paper PDF ↗ Code ↗ Hugging Face ↗ DOI ↗

Video demo

Real-time multi-view reconstruction

The demonstration shows HandMvNet reconstructing articulated 3D hand pose and mesh from synchronized camera views under hand-object interaction and occlusion.

Method

From view-specific evidence to one 3D hand

HandMvNet avoids volumetric reconstruction. It first samples image features at predicted 2D joints, fuses the resulting joint-aligned tokens across views, and directly regresses 21 joints and 778 mesh vertices.

HandMvNet architecture with pre-fusion, multi-view fusion, and post-fusion stages
HandMvNet pipeline. Shared image encoders produce joint-aligned features for every view. Cross-attention aggregates them in a target-view feature space before the joint and mesh decoder produces the final 3D reconstruction.
01

Joint-aligned sampling

Differentiable 2D joint estimates guide feature sampling, concentrating computation on hand structure rather than dense 3D volumes.

02

Cross-view attention

The target view acts as the query while features from the remaining cameras provide keys and values, integrating complementary evidence across viewpoints.

03

Joint & Mesh Decoding

A graph neural network decodes fused features into precise 3D joints, which pose and deform a hand template mesh to output the final hand shape.

Experiments

Accuracy and real-time efficiency

Across DexYCB-MV, HO3D-MV, and MVHand, HandMvNet reports the strongest root-relative joint and vertex errors among the compared approaches.

# Methods MPJPErel ↓ PAJ ↓ AUCJ@20 ↑ MPVPErel ↓ PAV ↓ AUCV@20 ↑
DexYCB-MV 1 📷 MvP 9.47 4.26 0.69 12.18 8.14 0.53
2 📷 PE-Mesh-TR 8.87 4.76 0.64 8.67 4.70 0.64
3 📷 FTL-Mesh-TR 9.81 5.51 0.59 9.80 5.75 0.59
4 📷 POEM 7.30 3.93 0.68 7.21 4.00 0.70
5 📷 Multi-view Fit. 8.77 5.19 0.65 8.71 5.29 0.65
6 📷 HandMvNet (ours) 6.73 4.08 0.67 7.19 4.52 0.65
7 📷 HandMvNet-HR (ours) 6.89 4.08 0.67 7.30 4.53 0.65
8 HandMvNet w/o cam. (ours) 7.03 4.13 0.66 7.38 4.56 0.64
9 HandMvNet-HR w/o cam. (ours) 7.28 4.20 0.65 7.62 4.69 0.63
HO3D-MV AUCJ@50 ↑ AUCV@50 ↑
10 📷 MvP 24.90 10.44 0.60 27.08 10.04 0.59
11 📷 PE-Mesh-TR 30.23 11.67 0.54 29.19 11.31 0.55
12 📷 FTL-Mesh-TR 34.74 10.72 0.52 33.53 10.56 0.53
13 📷 POEM 21.94 9.60 0.63 21.45 9.97 0.66
14 📷 HandMvNet (ours) 21.43 10.89 0.59 20.17 10.16 0.61
15 📷 HandMvNet-HR (ours) 20.73 11.01 0.61 19.82 10.73 0.62
16 HandMvNet w/o cam. (ours) 21.55 10.15 0.58 20.10 9.39 0.61
17 HandMvNet-HR w/o cam. (ours) 20.40 11.98 0.61 19.33 11.24 0.63
MVHand AUCJ@20 ↑ AUCV@20 ↑
18 📷 MediaPipe-DLT 17.24 9.97 0.28 18.42 7.74 0.21
19 📷 HandMvNet (ours) 2.07 1.30 0.90 7.57 4.14 0.62
20 📷 HandMvNet-HR (ours) 1.86 1.21 0.91 7.59 4.12 0.62
21 HandMvNet w/o cam. (ours) 2.05 1.28 0.90 7.62 4.11 0.62
22 HandMvNet-HR w/o cam. (ours) 1.77 1.14 0.91 7.46 4.15 0.63
Table 1: Quantitative results (mm) on the test sets of DexYCB-MV, HO3D-MV, and MVHand. 📷 denotes the methods that require camera parameters. The best and second-best results are highlighted in bold and underlined respectively.
Accuracy versus inference speed on DexYCB
Accuracy-speed trade-off. HandMvNet combines the lowest plotted DexYCB-MV error with the highest throughput among the compared methods.
Inference speed with two, four, six, and eight cameras
Scaling with camera views. Throughput decreases predictably as more views are added but remains real time at eight cameras.

DexYCB-MV

Eight-view hand-object interactions across ten subjects, evaluated on the official S0 split.

HO3D-MV

Five-view sequences with challenging object occlusions and a dedicated multi-view train/test split.

MVHand

Four-camera RGB-D hand poses spanning 21,200 synchronized multi-view frames.

Qualitative results

Reconstruction across datasets

The examples pair predicted joint overlays with reconstructed meshes across varied viewpoints, objects, and articulation. Select an image to view it at full resolution.

DexYCB-MV qualitative hand pose and mesh reconstruction resultsDexYCB-MV HO3D-MV qualitative hand pose and mesh reconstruction resultsHO3D-MV MVHand qualitative hand pose and mesh reconstruction resultsMVHand

Summary

Conclusion

We introduced HandMvNet, one of the first real-time methods for estimating 3D hand motion and shape from multi-view camera images. Our approach employs a multi-view attention-fusion mechanism that effectively integrates features from multiple viewpoints, delivering consistent and accurate absolute hand poses and shapes, free from the scale-depth ambiguities typically seen in monocular methods. Unlike previous multi-view approaches, HandMvNet eliminates the need for camera parameters to learn 3D geometry. We validated the architecture through extensive ablation studies and compared its performance with state-of-the-art methods. Experiments on public datasets demonstrate the effectiveness of our approach, delivering superior accuracy and inference speed compared to existing methods.

Reference

Citation

If this work is useful in your research, please cite the VISAPP paper:

@inproceedings{Ali2025,
  author    = {Ali, Muhammad Asad and Robertini, Nadia and Stricker, Didier},
  title     = {HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusion},
  booktitle = {Proceedings of the 20th International Joint Conference on Computer Vision,
               Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
  year      = {2025},
  pages     = {555--562},
  doi       = {10.5220/0013107300003912}
}