DREAM: Deep Learning-Based Recognition of Emotions from Multiple Affective Modalities Using Consumer-Grade Body Sensors and Video Cameras

Tools

Sharma, Aditi and Kumar, Akshi. 2024. DREAM: Deep Learning-Based Recognition of Emotions from Multiple Affective Modalities Using Consumer-Grade Body Sensors and Video Cameras. IEEE Transactions on Consumer Electronics, 70(1), pp. 1434-1442. ISSN 0098-3063 [Article]

Preview

Text
DREAM_Deep_Learning-Based_Recognition_of_Emotions_from_Multiple_Affective_Modalities_Using_Consumer-Grade_Body_Sensors_and_Video_Cameras (1).pdf - Accepted Version
Download (1MB) | Preview

Official URL: https://doi.org/10.1109/TCE.2023.3325317

Abstract or Description

Ambient smart cities exist on the intersection of digital technology, disruptive innovation and urban environments that now essentially augment affect empathy and intelligent interfacing for human computer interactions (HCI). This research puts forward a deep learning approach, DREAM, for recognition of emotions using three affective modalities (audio, video, physiological) to develop an empathetic HCI system using consumer electronic IoT sensors and cameras. Convolution network is used to train for physiological signals. VGG and ResNet have been used to pre-train the models for emotion recognition from video and audio signals. DREAM is then fine-tuned on the publicly available K-EmoCon dataset to accurately recognize emotion for each subject. K-EmoCon is annotated by seven persons for five discrete emotions, and two affect dimensions. Finally, a probability-based average decision-level fusion strategy is used for combining the outputs of all the modalities. Leave one out strategy is used to train and evaluate the model for subject specific accuracies. For discrete emotions highest accuracy of 81.7% and 82.4% is achieved for dimensional emotions. DREAM has performed better than existing state-of-the-art for both emotion models.

Item Type:

Article

Identification Number (DOI):

https://doi.org/10.1109/TCE.2023.3325317

Additional Information:

“© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.”

Keywords:

Facial Emotion Recognition, Human Computer Interaction, IoT sensors, K-EmoCon, Multi Modalities, Transfer Learning

Departments, Centres and Research Units:

Computing

Dates: