In a State, 2014
The interface entitled 'In a State' is a collaboration between artist Adinda van 't Klooster and composer/programmer Nick Collins. It captures live audio and uses machine learning to analyse the emotions expressed in improvised piano music, rendering live 3D animations based on drawings by the artist in response. An online research tool was used to help judge the effectiveness of certain graphics in terms of expressing particular emotions.
During the performance a computer agent improvises alongside the pianist following the emotions detected in the music. As the detected emotions change so will the animated graphics. The system is both aesthetically interesting and informative of musicians' expression of emotional state; it helps to highlight the problematic nature of emotion research and the ambiguities therein. A movie can be viewed here.
The first prototype of this live interactive performance was trialled at NIME2014 in London. Nick Collins improvised to the interface accompanied by a live electroacoustic agent. Both the pianists' and the agents' music were analysed for emotional expression and visualised with accompanying abstract animations. Computer agent and pianist tended to synchronise in terms of emotional expression though often not by mirroring the other's emotion exactly, but by approximating it. To read about the technical background of the system, see here.
More information can be found on the