Designing Gestures for Continuous Sonic Interaction

Tanaka, Atau; Di Donato, Balandino; Zbyszynski, Michael and Roks, Geert. 2019. 'Designing Gestures for Continuous Sonic Interaction'. In: The International Conference on New Interfaces for Musical Expression. Porto Alegre, Brazil 3-6 June 2019. [Conference or Workshop Item]

nime_19 (3).pdf - Published Version
Available under License Creative Commons Attribution.

Download (6MB) | Preview

Abstract or Description

We present a system that allows users to try different ways to train neural networks and temporal modelling to asso- ciate gestures with time-varying sound. We created a soft- ware framework for this and evaluated it in a workshop- based study. We build upon research in sound tracing and mapping-by-demonstration to ask participants to de- sign gestures for performing time-varying sounds using a multimodal, inertial measurement (IMU) and muscle sens- ing (EMG) device. We presented the user with two classical techniques from the literature, Static Position regression and Hidden Markov based temporal modelling, and pro- pose a new technique for capturing gesture anchor points on the fly as training data for neural network based regression, called Windowed Regression. Our results show trade- offs between accurate, predictable reproduction of source sounds and exploration of the gesture-sound space. Several users were attracted to our windowed regression technique. This paper will be of interest to musicians engaged in going from sound design to gesture design and offers a workflow for interactive machine learning.

Item Type:

Conference or Workshop Item (Paper)


Sonic Interaction Design, Interactive Machine Learning, Gestural Interaction

Related URLs:

Departments, Centres and Research Units:

Computing > Embodied AudioVisual Interaction Group (EAVI)


1 March 2019Submitted
20 March 2019Accepted
3 June 2019Published

Event Location:

Porto Alegre, Brazil

Date range:

3-6 June 2019

Item ID:


Date Deposited:

19 Jun 2019 13:07

Last Modified:

13 Jun 2021 11:13


View statistics for this item...

Edit Record Edit Record (login required)