IBM Cloud Instrument -Audio-Visual Exploration of Internet of Musical Things (IoMUST)

IBM Think 2019

IBM CLOUD INSTRUMENT

Powered by IBM Cloud™; the project aimed to create the new aesthetics of “connectedness” using multimedia and immersive experiences that offer the interconnected world of Internet of Things (IoT) and Artificial Intelligence (AI).

The Internet of Musical Things (IoMust) represents a coalescence of objects that share a collective “connectedness” in natural and human-made worlds.

As an interactive art-piece, the installation embodies movement, changing based on real-time data input.

INTERACTIVE AUDIO VISUAL INSTALLATION
​

The enablers of the Internet of Musical Things (IoMust) concept are powerful IoT technologies like IBM Watson supercomputer cluster, motion sensors, WiFi-enabled micro-controllers, LED displays, and back-end servers.

The spacial 3D-layer mapping framework and tools where connecting link to hand gestures and sound parameters. This gesture-instrument mapping system was later expanded to more audio-visual systems of manipulating input data data streams among diverse media systems like Notch and Touch Designer; heterogeneous interactive performance environments.

Through the complex network of technologies, we invite our audience to perform by controlling parameters of the analog synthesizer Moog Slim Phatty and observe the performance through audio-reactive visuals in real-time.

Through the complex network of technologies, we invite our audience to perform by controlling parameters of the analog synthesizer Moog Slim Phatty and observe the performance through audio-reactive visuals in real-time.

Another framework supporting sensor-based interactive performance is Wekinator and Leap Motion which seeks to extend music language toward gesture and visual languages, with a focus on analyzing expressive content in gesture and movement, and generating expressive outputs.

In particular, the sensor inputs of Leap Motion are fed to Deep Learning based AI and Machine Learning (ML) algorithms that are integrated in the IBM Cloud. Node-Red framework in the IBM Cloud allows us to create a connected network of art objects and human-computer interfaces, and multimedia frameworks such as Max/MSP, Notch, TouchDesigner, and custom software written in Python.

Max, also known as Max/MSP/Jitter, is a visual programming language for music and multimedia developed by San Francisco-based software company Cycling ’74.

EARLY TESTING WITH MOOG SYNTHESIZER

Early testing with Wekinator + Leap Motion

Before constructing the instrument as a system running on the cloud, I trained the artificial intelligence algorithm with 6 different presets consisting of 108 parameters on the Moog.

I created the necessary input data to play the instrument by transmitting the data of the x, y and z positions of my hand in the 3-dimensional environment to the algorithm through the leap motion sensor.

In this way, I created a 3-dimensional space that will affect the sound synthesis of the instrument.

MQTT | Web App

The custom web app also allows the audience to change the musics elements like BPM through the use of their phones.

MQTT MQ Telemetry Transport or Message Queuing Telemetry Transport) is an open OASIS and ISO standard (ISO/IEC 20922)[4] lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP; however, any network protocol that provides ordered, lossless, bi-directional connections can support MQTT. It is designed for connections with remote locations where a “small code footprint” is required or the network bandwidth is limited.