Unfolded MIRAGE
Year: 2020
Type: Interactive Installation
Tools: OpenGL, PyTorch, Touchdesigner, ZED Camera
Created by: Shaoyu Su
Documentation Video
System Pipeline
The system starts with volumetric capture of the exhibition zone using a stereo camera rig. A multi-layered customized OpenGL shader with buffer and feedback system processes the camera feed in depth, normal, point cloud, texture and Alberto channel into neural networking trained via PyTorch to visualize detected living being into abstract painterly forms. Instead of using commonly available vector-based transformation and displacement, a two-way shader is constantly monitoring each datapoint’s orientation in observing its optical attributes with consideration to its neighbors.
In the era deprived of privacy, people are subject to constantly being watched by the ubiquitous cameras. However, in this project the machine is the subject - it dehumanizes the exhibition space by deleting biological forms and translates them into pure data form of light and color. Capitalism and Centralism prompt the ubiquitous surveillance especially with cameras and machine vision - in public and on everyone’s smartphone - to gather bio-metric data as precise and representational as possible. Unfolded Mirage takes the completely opposite approach, seeking the beautiful sublime of visualization through disintegration.
User Interaction
Audiences can interact with this project if they were considered as "life forms" by the system which then tends to transform organic imagery into spatial-awarded data forms. As more and more audiences interact with Unfolded Mirage, the space builds up in a period of time via a customized buffer object, gradually triggering the transformation from 3-D point cloud of physical space into 4-D mathematical cyberspace (data-space). This is achieved by defining a group of complex numbers using quaternion so 4-D hyperplanes are formed. Different hyperplanes and axises will be blended, lerped and collapsed decided by computer vision, giving the 4-D cyber landscape an “unstable” look in constant motion. It also suggests the restless intelligent behavior from the machine's “cellular antenna” in its brain.
Mirage in nature is an optical phenomenon in which light rays bend via refraction to produce a displaced image of distant objects or the sky. In this installation, a cyber mirage is simulated through bending the long tunnel between seeing and knowing, human and machine into a mobius-ring with endless possibility. The system grows as more viewers and spaces it intakes via learning the 3-D environment’s features and classification through point cloud. The nature of its algorithmic structure is abundant with surprise and indeterminacy, revealing a novel machine spatial literacy, folding and unfolding the cross-dimensional sentience in the visual realm.
-
Special Thanks to dear friend Jordan Halsey