RF-based 3D skeletons

Mingmin Zhao, Yonglong Tian, Hang Zhao, Mohammad Abu Alsheikh, Tianhong Li, Rumen Hristov, Zachary Kabelac, Dina Katabi, Antonio Torralba

Research output: A Conference proceeding or a Chapter in BookConference contribution

23 Citations (Scopus)

Abstract

This paper introduces RF-Pose3D, the first system that infers 3D human skeletons from RF signals. It requires no sensors on the body, and works with multiple people and across walls and occlusions. Further, it generates dynamic skeletons that follow the people as they move, walk or sit. As such, RF-Pose3D provides a significant leap in RF-based sensing and enables new applications in gaming, healthcare, and smart homes. RF-Pose3D is based on a novel convolutional neural network (CNN) architecture that performs high-dimensional convolutions by decomposing them into low-dimensional operations. This property allows the network to efficiently condense the spatio-temporal information in RF signals. The network first zooms in on the individuals in the scene, and crops the RF signals reflected off each person. For each individual, it localizes and tracks their body parts - head, shoulders, arms, wrists, hip, knees, and feet. Our evaluation results show that RF-Pose3D tracks each keypoint on the human body with an average error of 4.2 cm, 4.0 cm, and 4.9 cm along the X, Y, and Z axes respectively. It maintains this accuracy even in the presence of multiple people, and in new environments that it has not seen in the training set. Demo videos are available at our website: http://rfpose3d.csail.mit.edu.

Original languageEnglish
Title of host publicationSIGCOMM 2018 - Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication
PublisherAssociation for Computing Machinery, Inc
Pages267-281
Number of pages15
ISBN (Electronic)9781450355674
ISBN (Print)9781450355674
DOIs
Publication statusPublished - 7 Aug 2018
Externally publishedYes
EventACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication -
Duration: 1 Jan 2011 → …

Publication series

NameSIGCOMM 2018 - Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication

Conference

ConferenceACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication
Abbreviated titleSIGCOMM
Period1/01/11 → …

    Fingerprint

Cite this

Zhao, M., Tian, Y., Zhao, H., Alsheikh, M. A., Li, T., Hristov, R., ... Torralba, A. (2018). RF-based 3D skeletons. In SIGCOMM 2018 - Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (pp. 267-281). (SIGCOMM 2018 - Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication). Association for Computing Machinery, Inc. https://doi.org/10.1145/3230543.3230579