Multi-modal fusion enhances activity recognition by integrating RGBD and skeletal data

Abstract

In this paper, the proposed work is based on a multi-modal fusion framework for human activity recognition (HAR). This approach makes use of three modalities such as RGB, depth maps and 3D-Skeletion joint position to develop robust HAR system. Two 3DCNN models with different network parameters and an LSTM model are used to obtain the features from each modality. Next, the score of each activity is obtained using SVM in each model and optimized using two evolutionally algorithms. The experimental work on the public dataset has also been discussed to validate the proposed approach. The experimental results show that the proposed framework is an improvement over previous work and is capable of accurately recognizing human activities