Tech Note: Handling Touch Data in Custom Applications

Waltz provides multiple paths for your custom application to receive touch data from any interactive surface supported in Waltz, including interactive LED floors. Custom applications may be your own graphics engine, a bespoke application for a single event, or a piece of creative programming software not otherwise covered by another tech note. For disguise media servers, including display of Notch content, see Handling Touch Data in disguise and Notch. For Touch Designer media projects, see Handling Touch Data in Touch Designer.

There are two different ways to handle touch data. The first, TUIO, provides a simple standard API for touch data that you may be able to use an existing library to handle. The second is via OSC, which provides more information about the lifecycle of each touch, but may require additional logic to handle touch tracking in your application.

Using TUIO to Handle Touch Data

TUIO is a protocol for communicating user interaction data like touches, object positions, and more over the network. TUIO is based on the OSC protocol, but defines the specific messages that are permitted. Waltz utilizes the 2dcur part of the TUIO specification to send information about touch data to remote machines as cursor events.

In the custom application

The easiest way to implement support for TUIO in your application is to use an existing library. For many common languages there are TUIO clients libraries that already exists that you can take advantage of, including for Java, C++, C#, JavaScript, Objective-C, and Swift. Please be aware that we are unable to provide any warranty or in depth support for these 3rd-party libraries, but are happy to assist in isolating issues as able.

These libraries will expose TUIO data is various ways, but at their cores they allow you to get the X and Y position of touch data in a way that can be tracked across frames to maintain a state for each event. For example, when someone touches a screen, a unique ID is given to that touch, and subsequent information about that touch, like a new location, will share that same ID until the person ceases to touch the screen.

Waltz uses the 2dcur data type for all touch data.

Your application will need to open a port on the local machine that will receive the TUIO data, and that same port, along with the machine's IP address, will need to be configured in Waltz as a destination for the touch data.

TUIO normalizes touch data between 0.0 and 1.0 in each dimension, and the bounds by which data is normalized is determined in Waltz. Typically, a 1920x1080 touch area will map zero to itself in each dimension, and 1.0 to 1920 for the x dimension and 1.0 to 1080 for the y dimension.

In Waltz

Use a TUIO 2D Cursor Output node, which takes a List<Touch> data, and provides those touches over the network as TUIO. The remote address should be configured to either point to specific machine you wish to receive TUIO, or a broadcast address for a group of machines to receive TUIO. The normalization dimensions should match the maximum range of touch data from the source of the touches. For LED floors, this will be the total pixel dimensions of the floor.

Using OSC to Handle Touch Data

OSC is a general-purpose protocol for transmitting show data over the network. Waltz is able to provide touch data over OSC through a combination of its Touch Mapper node and its OSC Output node. The Touch Mapper node creates the structured data from touches that will then be sent by the OSC Output node, without the need to manually map each piece of data for transmission.

In the custom application

Before you can begin handling the data Waltz provides, you first will need to be able to process OSC data in general. For many common languages there are OSC client libraries that already exist that you can take advantage of, including for Java, JavaScript, Objective-C, and Swift. Please be aware that we are unable to provide any warranty or in depth technical support for these 3rd-party libraries, but are happy to assist in isolating issues as able.

These libraries have various ways of allowing you to handle data that you will need to become familiar with. Waltz will send you updated data at the show's frame rate (typically 60FPS, but customizable in Waltz), and your application will need to handle the information as needed for your display purposes. Many aspects of the OSC Waltz provides can be customized, including the base part of the address path, the use of bundles, how multiple arguments are sent, and more. These instructions assume the default settings are used, but keep in mind settings in the Touch Mapper or the OSC Output node can modify this behavior. If you are interested, we highly suggest you familiarize yourself with the options available in those nodes to determine if any of them make sense for your situation.

By default, an OSC message is sent as part of a single OSC bundle per frame, with each message representing a single touch. Each message will have a unique OSC path, with a number on the end indicating which touch it is. If in a frame there are 8 touches active, these touches will be numbered 1 through 8. These numbers do not correlate to the sequentialId of the touch. A touch may have a different OSC path number every frame. Each of these messages will contain 11 arguments, defined as follows:

OSC Path: waltz/osc/touch1
#	Name	Type	Details
0	sequentialId	Integer	A unique ID for this touch. Each touch get a sequential, increasing, number. The first touch is always 1.
1	uuid	Integer	A unique ID for this touch, separate from the sequentialId. This number is a randomly generated unsigned integer.
2	isAlive	Integer	Always equals 1 if the touch is believed to be active, otherwise this value is 0. Dead touches only ever appear in a message if this node is configured with mapFixedNumber or maskDeadTouches enabled.
3	currentX	Float or Integer	The current x position of the touch. This argument is either an integer representing absolute pixels, or a float between 0.0 and 1.0. This is dependent on whether normalizePosition is enabled.
4	currentY	Float or Integer	The current y position of the touch. This argument is either an integer representing absolute pixels, or a float between 0.0 and 1.0. This is dependent on whether normalizePosition is enabled.
5	initialX	Float or Integer	The initial x position of the touch. This argument is either an integer representing absolute pixels, or a float between 0.0 and 1.0. This is dependent on whether normalizePosition is enabled.
6	initialY	Float or Integer	The initial y position of the touch. This argument is either an integer representing absolute pixels, or a float between 0.0 and 1.0. This is dependent on whether normalizePosition is enabled.
7	radius	Integer	The calculated radius of the touch, in pixels.
8	creationMillis	Integer	The time, in milliseconds, at which the touch was created. See the heartbeat message for the current time.
9	lastUpdatedMillis	Integer	The time, in milliseconds, at which the touch was last updated to reflect a change in position, radius, or continued existence. See the heartbeat message for the current time.
10	ageMillis	Integer	The age of the touch, in milliseconds.

Additionally, two other messages are sent, one with the current time and one with the number of touches to be expected in the current frame. These messages are formatted as follows:

OSC Path: waltz/osc/heartbeat
#	Name	Type	Details
0	heartbeat	Integer	The current time, in milliseconds. This number will either be relative to the opening of the current show file, or relative to Epoch time, depending on whether useEpochTime is enabled. Should the need arise, this number will loop around to 0 after reaching the maximum value for an integer, which would occur after 49 days of continuous operation. For long-term installations it is recommended that this situation be handled gracefully by the receiver of this data.

OSC Path: waltz/osc/count
#	Name	Type	Details
0	count	Integer	The number of touches being sent in the current frame. This number will be 0 if no touches are currently being sent.

In Waltz

Use a Touch Mapper node, which accepts a List<Touch> and provides a Map of information ready to be sent over OSC with an OSC Output node. The remote address should be configured to either point to specific machine you wish to receive the OSC, or a broadcast address for a group of machines to receive the OSC. Keep in mind that both nodes contain settings that can affect the format of the OSC data being transmitted.