Through-Wall Human Pose Estimation Using Radio Signals

Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, Dina Katabi

Research output: A Conference proceeding or a Chapter in BookConference contribution

23 Citations (Scopus)

Abstract

This paper demonstrates accurate human pose estimation through walls and occlusions. We leverage the fact that wireless signals in the WiFi frequencies traverse walls and reflect off the human body. We introduce a deep neural network approach that parses such radio signals to estimate 2D poses. Since humans cannot annotate radio signals, we use state-of-the-art vision model to provide cross-modal supervision. Specifically, during training the system uses synchronized wireless and visual inputs, extracts pose information from the visual stream, and uses it to guide the training process. Once trained, the network uses only the wireless signal for pose estimation. We show that, when tested on visible scenes, the radio-based system is almost as accurate as the vision-based system used to train it. Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios. Demo videos are available at our website.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages7356-7365
Number of pages10
ISBN (Electronic)9781538664209
ISBN (Print)9781538664216
DOIs
Publication statusPublished - 16 Jun 2018
Externally publishedYes
Event31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 - Salt Lake City, United States
Duration: 18 Jun 201822 Jun 2018

Conference

Conference31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
CountryUnited States
CitySalt Lake City
Period18/06/1822/06/18

Fingerprint

Websites
Deep neural networks

Cite this

Zhao, M., Li, T., Alsheikh, M. A., Tian, Y., Zhao, H., Torralba, A., & Katabi, D. (2018). Through-Wall Human Pose Estimation Using Radio Signals. In Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 (pp. 7356-7365). [8578866] IEEE, Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/CVPR.2018.00768
Zhao, Mingmin ; Li, Tianhong ; Alsheikh, Mohammad Abu ; Tian, Yonglong ; Zhao, Hang ; Torralba, Antonio ; Katabi, Dina. / Through-Wall Human Pose Estimation Using Radio Signals. Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018. IEEE, Institute of Electrical and Electronics Engineers, 2018. pp. 7356-7365
@inproceedings{681fcb6381004d979d139fd207ff851f,
title = "Through-Wall Human Pose Estimation Using Radio Signals",
abstract = "This paper demonstrates accurate human pose estimation through walls and occlusions. We leverage the fact that wireless signals in the WiFi frequencies traverse walls and reflect off the human body. We introduce a deep neural network approach that parses such radio signals to estimate 2D poses. Since humans cannot annotate radio signals, we use state-of-the-art vision model to provide cross-modal supervision. Specifically, during training the system uses synchronized wireless and visual inputs, extracts pose information from the visual stream, and uses it to guide the training process. Once trained, the network uses only the wireless signal for pose estimation. We show that, when tested on visible scenes, the radio-based system is almost as accurate as the vision-based system used to train it. Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios. Demo videos are available at our website.",
keywords = "Human pose estimation, radio signals, WiFi",
author = "Mingmin Zhao and Tianhong Li and Alsheikh, {Mohammad Abu} and Yonglong Tian and Hang Zhao and Antonio Torralba and Dina Katabi",
year = "2018",
month = "6",
day = "16",
doi = "10.1109/CVPR.2018.00768",
language = "English",
isbn = "9781538664216",
pages = "7356--7365",
booktitle = "Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
address = "United States",

}

Zhao, M, Li, T, Alsheikh, MA, Tian, Y, Zhao, H, Torralba, A & Katabi, D 2018, Through-Wall Human Pose Estimation Using Radio Signals. in Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018., 8578866, IEEE, Institute of Electrical and Electronics Engineers, pp. 7356-7365, 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, United States, 18/06/18. https://doi.org/10.1109/CVPR.2018.00768

Through-Wall Human Pose Estimation Using Radio Signals. / Zhao, Mingmin; Li, Tianhong; Alsheikh, Mohammad Abu; Tian, Yonglong; Zhao, Hang; Torralba, Antonio; Katabi, Dina.

Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018. IEEE, Institute of Electrical and Electronics Engineers, 2018. p. 7356-7365 8578866.

Research output: A Conference proceeding or a Chapter in BookConference contribution

TY - GEN

T1 - Through-Wall Human Pose Estimation Using Radio Signals

AU - Zhao, Mingmin

AU - Li, Tianhong

AU - Alsheikh, Mohammad Abu

AU - Tian, Yonglong

AU - Zhao, Hang

AU - Torralba, Antonio

AU - Katabi, Dina

PY - 2018/6/16

Y1 - 2018/6/16

N2 - This paper demonstrates accurate human pose estimation through walls and occlusions. We leverage the fact that wireless signals in the WiFi frequencies traverse walls and reflect off the human body. We introduce a deep neural network approach that parses such radio signals to estimate 2D poses. Since humans cannot annotate radio signals, we use state-of-the-art vision model to provide cross-modal supervision. Specifically, during training the system uses synchronized wireless and visual inputs, extracts pose information from the visual stream, and uses it to guide the training process. Once trained, the network uses only the wireless signal for pose estimation. We show that, when tested on visible scenes, the radio-based system is almost as accurate as the vision-based system used to train it. Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios. Demo videos are available at our website.

AB - This paper demonstrates accurate human pose estimation through walls and occlusions. We leverage the fact that wireless signals in the WiFi frequencies traverse walls and reflect off the human body. We introduce a deep neural network approach that parses such radio signals to estimate 2D poses. Since humans cannot annotate radio signals, we use state-of-the-art vision model to provide cross-modal supervision. Specifically, during training the system uses synchronized wireless and visual inputs, extracts pose information from the visual stream, and uses it to guide the training process. Once trained, the network uses only the wireless signal for pose estimation. We show that, when tested on visible scenes, the radio-based system is almost as accurate as the vision-based system used to train it. Yet, unlike vision-based pose estimation, the radio-based system can estimate 2D poses through walls despite never trained on such scenarios. Demo videos are available at our website.

KW - Human pose estimation

KW - radio signals

KW - WiFi

UR - http://www.scopus.com/inward/record.url?scp=85055086378&partnerID=8YFLogxK

UR - http://openaccess.thecvf.com/content_cvpr_2018/CameraReady/2406.pdf

U2 - 10.1109/CVPR.2018.00768

DO - 10.1109/CVPR.2018.00768

M3 - Conference contribution

SN - 9781538664216

SP - 7356

EP - 7365

BT - Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018

PB - IEEE, Institute of Electrical and Electronics Engineers

ER -

Zhao M, Li T, Alsheikh MA, Tian Y, Zhao H, Torralba A et al. Through-Wall Human Pose Estimation Using Radio Signals. In Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018. IEEE, Institute of Electrical and Electronics Engineers. 2018. p. 7356-7365. 8578866 https://doi.org/10.1109/CVPR.2018.00768