Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers

ECCV 2022

Junhyeong Cho, Kim Youwang, Tae-Hyun Oh

Pohang University of Science and Technology (POSTECH)

Paper & GitHub

Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers

Junhyeong Cho, Kim Youwang, Tae-Hyun Oh

European Conference on Computer Vision (ECCV), 2022

[Paper] [GitHub] [Bibtex]


Transformer encoder architectures have recently achieved state-of-the-art results on monocular 3D human mesh reconstruction, but they require a substantial number of parameters and expensive computations. Due to the large memory overhead and slow inference speed, it is difficult to deploy such models for practical use. In this paper, we propose a novel transformer encoder-decoder architecture for 3D human mesh reconstruction from a single image, called FastMETRO. We identify the performance bottleneck in the encoder-based transformers is caused by the token design which introduces high complexity interactions among input tokens. We disentangle the interactions via an encoder-decoder architecture, which allows our model to demand much fewer parameters and shorter inference time. In addition, we impose the prior knowledge of human body's morphological relationship via attention masking and mesh upsampling operations, which leads to faster convergence with higher accuracy. Our FastMETRO improves the Pareto-front of accuracy and efficiency, and clearly outperforms image-based methods on Human3.6M and 3DPW. Furthermore, we validate its generalizability on FreiHAND.


Experimental Results


This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00290, Visual Intelligence for Space-Time Understanding and Generation based on Multi-layered Visual Common Sense; and No. 2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH)).

This template was originally made by Phillip Isola and Richard Zhang for a ECCV project; template code can be found here.