Creating immersive audio-visual experiences

Audiences in China, the UK, Singapore and Canada are experiencing innovative immersive audio-visual performance, thanks to a research project by Dr Stephen Gibson at Northumbria University. Virtual VJ, uses motion-tracking technology to generate new forms of artistic expression using body motion to control the behaviour of an audio and video environment.

VJ creates the opportunity for a high degree of audience participation, challenging the sensory experiences of the audiences. During 2011, Gibson exhibited Virtual VJ to audiences made up of professional designers, artists and academics, political figures and high profile business people at venues in Vancouver, Canada, and at the Jade Valley in X’ian, China. Audience members were invited to perform using the motion-trackers and gain a direct sense of how motion-tracking can be used to control simultaneous media with their bodies. This exposed them to innovative ideas about art and interactive experiences, especially participative art forms.

Virtual VJ is technologically possible through using 3D motion-tracking hardware and software - The Gesture and Media System (GAMS), produced by Apr Inc. and Limbic Media Corporation. GAMS allows artists and other users to ‘map’ an interactive space with sound, light and images, with the user-movement controlling these elements via hand-held trackers. There is increasing interest across the field of Human-Computer Interaction (HCI) in the responsiveness of technology to dynamic embodied human use and vice versa.

In Singapore, the project has influenced the design of Value Lab Asia. Value Lab Asia, is a high-tech environment opened in 2013 and located at the Singapore National Research Foundation. It is equipped with several touch surfaces and a very high-resolution video wall. It is used for a wide range of applications such as participatory planning and design, information visualisation and discovery and remote conferencing. After experiencing Virtual VJ at Singapore’s Digital Art Weeks, Value Lab Asia is planning to include a similar audio-visual motion tracking system in an upcoming development phase.

This is confirmed by Stefan Müller Arisona, Principal Investigator at Value Lab Asia: “Working with a motion-tracking system for physical and gestural live visuals control has helped to add a new scale of dynamics to live animated content, making content itself an immersive gesture, similar to a lighting system and disconnected from projection screen.”