Bimodal Features Depth maps and 3D Skeleton Information with Time Sequence for HAR

Abstract

Over the past few decades, as software, hardware, and big data of computers improve by leaps and bounds, deep learning has been applied in all aspects of artificial intelligence in the real world, such as detection, segmentation and classification. The traditional methodology is to design a hand-crafted feature descriptor to represent an object for human activity recognition. The deep learning-based approaches directly extract features map based on the neural network processing. Compared with RGB video, depth sequences are more advantageous in lighting changes and information collection. In this work, we study depth maps based on deep learning and corresponding skeleton joint information. The 3D Convolutional Neural Network (CNN) directly learns spatial-temporal feature maps from depth map sequences. We extract the skeleton joints information where skeleton joints point distance, angle information and time sequences of skeleton joints point changes in different frames to classify human activities with the spatial-temporal feature map. We validate the hybrid feature vector with the Support Vector Machines (SVM) classifier on the MSR-Action3D and the UTKinect-Action3D benchmark datasets. Our proposed method proves that it has very promising classification accuracy compared with the current experiment result.