Multimodal recognition of reading activity in transit using body-worn sensors

Tools

Bulling, Andreas; Ward, Jamie A and Gellersen, Hans. 2012. Multimodal recognition of reading activity in transit using body-worn sensors. ACM Transactions on Applied Perception, 9(1), 2. ISSN 1544-3558 [Article]

No full text available

Text
bulling12_tap.pdf - Published Version
Permissions: Administrator Access Only
Download (5MB)

Official URL: https://dl.acm.org/doi/10.1145/2134203.2134205

Abstract or Description

Reading is one of the most well studied visual activities. Vision research traditionally focuses on understanding the perceptual and cognitive processes involved in reading. In this work we recognise reading activity by jointly analysing eye and head movements of people in an everyday environment. Eye movements are recorded using an electrooculography (EOG) system; body movements using body-worn inertial measurement units. We compare two approaches for continuous recognition of reading: String matching (STR) that explicitly models the characteristic horizontal saccades during reading, and a support vector machine (SVM) that relies on 90 eye movement features extracted from the eye movement data. We evaluate both methods in a study performed with eight participants reading while sitting at a desk, standing, walking indoors and outdoors, and riding a tram. We introduce a method to segment reading activity by exploiting the sensorimotor coordination of eye and head movements during reading. Using person-independent training, we obtain an average precision for recognising reading of 88.9% (recall 72.3%) using STR and of 87.7% (recall 87.9%) using SVM over all participants. We show that the proposed segmentation scheme improves the performance of recognising reading events by more than 24%. Our work demonstrates that the joint analysis of multiple modalities is beneﬁcial for reading recognition and opens up discussion on the wider applicability of this recognition approach to other visual and physical activities.

Item Type:

Article

Identification Number (DOI):

https://doi.org/10.1145/2134203.2134205

Keywords:

Algorithms, Experimentation, Measurement, Recognition of reading, eye movement analysis, multimodal sensing, sensorimotor coordi- nation, head movements, electrooculography (EOG)

Departments, Centres and Research Units:

Computing

Dates: