Ensembles of Bidirectional LSTM and GRU Neural Nets for Predicting Mother-Infant Synchrony in Videos

Tools

Stamate, Daniel; Davuloori, Pradyumna; Logofatu, Doina; Mercure, Evelyne; Addyman, Caspar and Tomlinson, Mark. 2024. 'Ensembles of Bidirectional LSTM and GRU Neural Nets for Predicting Mother-Infant Synchrony in Videos'. In: Engineering Applications of Neural Networks: 25th International Conference on Engineering Applications of Neural Networks (EANN 2024). Corfu, Greece 27 - 30 June 2024. [Conference or Workshop Item]

No full text available

Text
eann_stamate_etal.pdf - Accepted Version
Permissions: Administrator Access Only until 22 June 2025.
Download (1MB)

Official URL: https://doi.org/10.1007/978-3-031-62495-7_25

Abstract or Description

The importance of positive, healthy and reciprocal interactions between mother and infant cannot be understated as it leaves a lasting impact on the rest of the infant’s life. One way to identify a positive interaction between two people is the amount of nonverbal synchrony - or spontaneous coordination of bodily movements, present in the interaction. This work proposes a neural network and ensemble learning based approach to automatically labelling a mother-infant dyad interaction as high versus low by predicting the level of synchrony of the interaction. Bidirectional Long Short-Term Memory (BiLSTM) and Bidirectional Gated Recurrent Unit (BiGRU) models were trained and evaluated on a dataset consisting of 25 key body position coordinates of mother and infant extracted with an AI specialised tool called OpenPose, from 58 different videos. Ensembles of 30 such bidirectional recurrent neural network base models were built and then post-processed via ROC analysis, to improve prediction stability and performance, both of which assessed in a Monte Carlo validation procedure of 30 iterations. The prediction performances on the unseen test samples for the ensembles of BiLSTM and ensembles of BiGRU models include a mean AUC of 0.781 and 0.796, a mean precision of 0.857 and 0.899, and a mean specificity of 0.817 and 0.872, respectively. In particular our models predict higher probability scores for the high synchrony class versus the low synchrony class in 80% of cases. Moreover the achieved high precision level indicates that 90% of mother-infant dyads predicted to be in the high synchrony class are predicted correctly, and the high specificity level indicates a detection rate of the mother-infant dyads with low synchrony in 87% of cases, suggesting these models’ high capability for automatically flagging cases that may be clinically relevant for further investigation and potential intervention.

Item Type:

Conference or Workshop Item (Paper)

Identification Number (DOI):

https://doi.org/10.1007/978-3-031-62495-7_25

Keywords:

Mother-infant synchrony detection, Recurrent neural networks, Bidirectional LSTM, Bidirectional GRU, Ensemble learning, Model postprocessing optimisation, Monte Carlo validation, Video classification

Departments, Centres and Research Units:

Computing
Psychology

Dates: