The AudioVirtualizer is an interactive and generative Virtual Reality environment created as a VR art experience utilizing the HTC Vive Pro and Oculus Rift headsets. A participant explores three different Op Art inspired scenes which reveal different mappings between musical input and visual output. In each scene the user can select from a range of different soundtracks, or live microphone input, to investigate how the visuals respond to the sound. The graphics are based on different drawings and 3D models by Adinda van ’t Klooster, and teleportation between the scenes as well as a within scene navigation system allow the user to explore each environment to the full. The audio engine tracks low- as well as mid-level features in the sound, allowing more complicated visual returns than previous FFT filterbank based audiovisualisers.
Our AudioVirtualizer system first launched at PROTO in September 2019, was created for SteamVR and currently runs as a standalone application generated through Unity for the Vive Pro and Oculus Rift VR headsets. For development we worked with an NVIDIA GForceGTX1070 graphics card and Intel Core i5 system. The graphics are generated in response to changes in low- and mid-level sound features and also exist in the ‘landscapes’ already via skyboxes that contain high-resolution scanned drawings. Certain elements from the drawings are then repeated in sculptural objects that are generated in response to the sound and move through the landscape in different ways. The drawings have a sense of minimalism and are manipulated digitally to continue their endlessly repeating patterns in the 3D space as seamlessly as possible. The user is able to see how the graphics respond to different sound sources through a built-in menu option that allows them to choose one of four soundtracks or live microphone input. Pressing the menu button on the controller overlays the scene with five sculptural objects, each of these leading to a different sound source when pressed. Selecting the menu button again will remove the options and continue to play the current selection. All graphics and system design are by Adinda van ’t Klooster, machine-listening tech and some Unity programming was due to Nick Collins, with additional assistance with Unity VR programming provided by Nathan Flounders.
There are essentially three different scenes and an entrance space where navigation can be practiced. Using the large circular touch pad on the VIVE controller one can move forwards (up), to the right (right), the left (left) and backwards (down). Going up and down in the 3D space is achieved by pressing the trigger as well as keeping the finger on the up or down position of the circular dial. For non-gamers this takes some practice so this is done in an antechamber space without animated graphics. One can be teleported into the next scene by navigating to a red dildo-like sculpture and selecting it.