Dobre, Georgiana Cristina; Gillies, Marco and Pan, Xueni. 2022. Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games. Virtual Reality, 26(4), pp. 1519-1538. ISSN 1359-4338 [Article]

Preview

Text Dobre2022_Article_ImmersiveMachineLearningForSoc.pdf - Published Version Available under License Creative Commons Attribution. Download (1MB) | Preview

Abstract or Description

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g., head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalise it for detecting human-defined social attitudes.

Item Type:	Article
Identification Number (DOI):	https://doi.org/10.1007/s10055-022-00644-4
Additional Information:	We work on this project in collaboration with two game studios: Dream Reality Interactive (https://www.dreamrealityinteractive.com/) and Maze Theory (https://www.maze-theory.com/); the project being supported by Innovate UK grant TS/S02221X/1. This work was also partly supported by grant EP/L015846/1 for the Centre for Doctoral Training in Intelligent Games and Game Intelligence (http://www.iggi.org.uk/) from the UK Engineering and Physical Sciences Research Council (EPSRC). The online version of the article contains supplementary material available at https://doi.org/10.1007/s10055-022-00644-4
Keywords:	Artificial Intelligence, Expressive Body Language, Gaming, Human-Computer Interaction, Virtual Agents, Virtual Reality
Departments, Centres and Research Units:	Computing
Dates:	Date Event 12 February 2022 Accepted 7 April 2022 Published Online December 2022 Published
Item ID:	31463
Date Deposited:	15 Feb 2022 09:29
Last Modified:	24 Nov 2022 11:40
Peer Reviewed:	Yes, this version has been peer-reviewed.
URI:	https://research.gold.ac.uk/id/eprint/31463

View statistics for this item...

Edit Record (login required)