Sculpting Unrealities: Using Machine Learning to Control Audiovisual Compositions in Virtual Reality
Tools
Tools
Dunphy, Bryan. 2022. Sculpting Unrealities: Using Machine Learning to Control Audiovisual Compositions in Virtual Reality. Doctoral thesis, Goldsmiths, University of London [Thesis]
|
Text (Sculpting Unrealities: Using Machine Learning to Control Audiovisual Compositions in Virtual Reality)
COM_thesis_DunphyB_2022.pdf - Accepted Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (3MB) | Preview |
Abstract or Description
This thesis explores the use of interactive machine learning (IML) techniques to control audiovisual compositions within the emerging medium of virtual reality (VR). Accompanying the text is a portfolio of original compositions and open-source software. These research outputs represent the practical elements of the project that help to shed light on the core research question: how can IML techniques be used to control audiovisual compositions in VR? In order to find some answers to this question, it was broken down into its constituent elements. To situate the research, an exploration of the contemporary field of audiovisual art locates the practice between the areas of visual music and generative AV. This exploration of the field results in a new method of categorising the constituent practices. The practice of audiovisual composition is then explored, focusing on the concept of equality. It is found that, throughout the literature, audiovisual artists aim to treat audio and visual material equally. This is interpreted as a desire for balance between the audio and visual material. This concept is then examined in the context of VR. A feeling of presence is found to be central to this new medium and is identified as an important consideration for the audiovisual composer in addition to the senses of sight and sound. Several new terms are formulated which provide the means by which the compositions within the portfolio are analysed. A control system, based on IML techniques, is developed called the Neural AV Mapper. This is used to develop a compositional methodology through the creation of several studies. The outcomes from these studies are incorporated into two live performance pieces, Ventriloquy I and Ventriloquy II. These pieces showcase the use of IML techniques to control audiovisual compositions in a live performance context. The lessons learned from these pieces are incorporated into the development of the ImmersAV toolkit. This open-source software toolkit was built specifically to allow for the exploration of the IML control paradigm within VR. The toolkit provides the means by which the immersive audiovisual compositions, Obj_#3 and Ag Fás Ar Ais Arís are created. Obj_#3 takes the form of an immersive audiovisual sculpture that can be manipulated in real-time by the user. The title of the thesis references the physical act of sculpting audiovisual material. It also refers to the ability of VR to create alternate realities that are not bound to the physics of real-life. This exploration of unrealities emerges as an important aspect of the medium. The final piece in the portfolio, Ag Fás Ar Ais Arís takes the knowledge gained from the earlier work and pushes the boundaries to maximise the potential of the medium and the material.
|
Item Type: |
Thesis (Doctoral) |
| Identification Number (DOI): |
|
| Additional Information: |
To request a download link for the full media pack (12.5GB), please email bryandunphy@gmail.com. |
| Keywords: |
audiovisual art; audiovisual composition; machine learning; virtual reality; immersion; presence; live performance; generative art; real-time graphics; audio synthesis; visual music; generative AV |
| Departments, Centres and Research Units: |
|
| Date: |
30 September 2022 |
| Item ID: |
32306 |
| Date Deposited: |
17 Oct 2022 10:29 |
| Last Modified: |
25 Oct 2022 16:31 |
|
URI: |
View statistics for this item...
 |
Edit Record (login required) |