Mienophone is created by Zhe Wang and Shang Heitz in collaboration, since 2019.
Music is controlling our emotions. It sometimes feels a bit like being a string puppet, while music is the puppet master. Music can make us dance or sink into sadness.
As Machine Learning Algorithms are becoming better at reading human emotions, can we use this technology to reverse this power of control? How does music sound like if it was directly controlled by our expressions and not the other way around? How would we react? Would it perhaps reinforce the emotion? Can technology help us to become more aware and better understand our emotions? And how does all this challenge the purpose of art and the role of the artist in the future?
These are the questions the artists explore with the Mienophone. In the exhibition, the artist duo is having a dialogue through different media: the Mienophone installation, video works and photos.
The project is funded by KARIN ABT-STRAUBINGER Stiftung
Technical support: Berlin Glassworks
About “Mienophone”
Mienophone is an installation consisting of a glass sculpture and a software synthesizer developed in MAX/NodeJS. A camera is pointing at the viewer. Selected frames of the video stream are transmitted to the Microsoft Cognitive Services Vision API. The Machine Learning Algorithm returns a numeric value for a set of emotions detected in the viewer’s face. The Algorithm can detttect anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise. Each value is used to control a set of parameters of the software synthesizer and allows the viewer to influence the sound by expressing emotions.
The camera is placed in an organic-shaped glass sculpture. Since the glass surface is not flat, the camera’s line of sight will be distorted, so the Machine Learning Algorithm will also read the viewer’s image distorted. This also reminds us that whether Machine Learning Algorithms can see us clearly, does AI and humans really “see” each other? Is the “seeing” equal?
The shape of the glass is symbolically presented in a form similar to a human face; just like our imagination of AI, it is a human-like, inhuman subject. When faced with this infant-sized, fragile, human-like device, we will regard ourselves as parents of this new intelligence.
The main element that composes glass is silicide, and computer chips are the same. The history of glassmaking can be traced back to BC, far before the invention of iron melting technology. Although computer chips have only be invented for a few decades, the powerful driving force they provide to the development of human society cannot be quantified.
The development of Machine Learning Algorithm has been closely linked to the future of humankind. In the most optimistic scenario, we may be able to create a utopia where humans and superintelligence can coexist peacefully. Or, the omnipotent AI has mastered higher intelligence than human beings. Human beings live in a society controlled by AI, lose their creative ability, and only lament their own destiny.
Videos
In “once I showed you what I want,” Zhé deconstructed the performances, images, and emotions of actors in movies by reinterpreting the close-ups of performers in old movies. Emotions are expressed through facial expressions, often in a moment, but at this moment, tens of thousands of facial nerves drive muscles to move together, just like a feast. Sometimes the emotions expressed by people are also extremely complex, such as self-deprecating laughter under extreme sadness. Can AI read such complex emotions instead of just telling us that this expression is 99.9% happy?
In "let's laugh together," Zhé focused on the expression of laughter. In Dialectic of Enlightenment Adorno's description of laughter is very touché,
"Conciliatory laughter is heard as the echo of an escape from power; the wrong kind overcomes fear by capitulating to the forces which are to be feared. It is the echo of power as something inescapable."
In our power interaction with others, laughter is produced. We laugh at our weaknesses, we laugh at our ignorance, we laugh at the laughs of others.