Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games

Dobre, Georgiana Cristina; Gillies, Marco and Pan, Xueni. 2022. Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games. Virtual Reality, 26(4), pp. 1519-1538. ISSN 1359-4338 [Article]

Dobre2022_Article_ImmersiveMachineLearningForSoc.pdf - Published Version
Available under License Creative Commons Attribution.

Download (1MB) | Preview

Abstract or Description

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g., head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.

Item Type:


Identification Number (DOI):

Additional Information:

We work on this project in collaboration with two game studios: Dream Reality Interactive ( and Maze Theory (; the project being supported by Innovate UK grant TS/S02221X/1. This work was also partly supported by grant EP/L015846/1 for the Centre for Doctoral Training in Intelligent Games and Game Intelligence ( from the UK Engineering and Physical Sciences Research Council (EPSRC).

The online version of the article contains supplementary material available at


Artificial Intelligence, Expressive Body Language, Gaming, Human-Computer Interaction, Virtual Agents, Virtual Reality

Departments, Centres and Research Units:



12 February 2022Accepted
7 April 2022Published Online
December 2022Published

Item ID:


Date Deposited:

15 Feb 2022 09:29

Last Modified:

24 Nov 2022 11:40

Peer Reviewed:

Yes, this version has been peer-reviewed.


View statistics for this item...

Edit Record Edit Record (login required)