An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound

Tools

Tanaka, Atau; Visi, Federico; Donato, Balandino Di; Klang, Martin and Zbyszyński, Michael. 2023. An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound. Computer Music Journal, 47(1), pp. 64-84. ISSN 0148-9267 [Article]

Preview

Text
comj_a_00672.pdf - Published Version
Available under License Creative Commons Attribution.
Download (1MB) | Preview

Official URL: https://doi.org/10.1162/comj_a_00672

Abstract or Description

This article presents a custom system combining hardware and software that senses physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning–based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performances by a range of musical practitioners.

Item Type:

Article

Identification Number (DOI):

https://doi.org/10.1162/comj_a_00672

Additional Information:

The research leading to these results has received funding from the European Research Council under the European Union's Seventh Framework Programme (FP/2007-2013) / ERC Grant FP7-283771. It has also received funding from the Horizon 2020 research and innovation programme, grant agreement no. 789,825. Continuing work is supported by the French Agence Nationale de la Recherche ANR-21-CE38-0018. We would like to thank the participants in our user trials.