Open Access Repository

Deep auto-encoders with sequential learning for multimodal dimensional emotion recognition


Downloads per month over past year

Nguyen, D, Nguyen, DT, Zeng, R, Nguyen, TT, Tran, S ORCID: 0000-0002-5912-293X, Nguyen, TK, Sridharan, S and Fookes, C 2021 , 'Deep auto-encoders with sequential learning for multimodal dimensional emotion recognition' , IEEE Transactions on Multimedia , pp. 1-12 , doi: 10.1109/TMM.2021.3063612.

146291 - Deep a...pdf | Download (4MB)

| Preview


Multimodal dimensional emotion recognition has drawn a great attention from the affective computing community and numerous schemes have been extensively investigated, making a significant progress in this area. However, several questions still remain unanswered for most of existing approaches including: (i) how to simultaneously learn compact yet representative features from multimodal data, (ii) how to effectively capture complementary features from multimodal streams, and (iii) how to perform all the tasks in an end-to-end manner. To address these challenges, in this paper, we propose a novel deep neural network architecture consisting of a two-stream auto-encoder and a long short term memory for effectively integrating visual and audio signal streams for emotion recognition. To validate the robustness of our proposed architecture, we carry out extensive experiments on the multimodal emotion in the wild dataset: RECOLA. Experimental results show that the proposed method achieves state-of-the-art recognition performance.

Item Type: Article
Authors/Creators:Nguyen, D and Nguyen, DT and Zeng, R and Nguyen, TT and Tran, S and Nguyen, TK and Sridharan, S and Fookes, C
Keywords: multimodal emotion recognition, dimensional emotion recognition, auto-encoder, long short term memory, deep learning
Journal or Publication Title: IEEE Transactions on Multimedia
Publisher: Ieee-Inst Electrical Electronics Engineers Inc
ISSN: 1520-9210
DOI / ID Number: 10.1109/TMM.2021.3063612
Copyright Information:

Copyright 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Item Statistics: View statistics for this item

Actions (login required)

Item Control Page Item Control Page