substrate is a Processing library developed to prototype gestural interfaces with mobile phones. The library tracks and moves a collection of pointers on a screen as they are moved and controlled by the movement of mobile devices, through an application built on Oblong’s g-speak platform. Colleagues at Oblong, Zai Chang and Alex Valli, wrote an iPhone application that connects to a server and streams user interaction data, including readings from the phone’s accelerometer. The Processing library (which I co-developed with David Wicks) receives the data and uses it to control the data for a pointer. Multiple pointers can be added to an application if multiple phones are connected.

The video above shows a demo of the technology in action. The user moves a cursor by tapping on the phone screen and moving the device. As the cursor moves, the grid responds by revealing the identity of squares near the cursor. A double tap reveals the identity of all squares in the grid.

The library was used for a workshop at Eyeo 2012 with Oblong Industries.