[Conn-Dev Final] DDRPi: A DDR game with lights and sound (Raspberry Pi + MQTT + TouchDesigner)

Dance Dance R-Pi with @Nick Boss

Demo of Shin dance-testing our rough prototype.

Last week, Nick and I figured out how to use DMX lights over USB connection with QLC+. We finally agreed on using TouchDesigner (for general game flow, kinect input, and light output) and a Raspberry Pi Zero W (for button interface and calculating the score) as our tools of choice. Though we experimented with HTTP as a possible communication protocol, and Node Red for flow editing, eventually we ended up on using MQTT in TouchDesigner directly. There are many more things to figure out, and we would tackle them one by one:

1) Using the existing light system over Ethernet with TouchDesigner. 2) Building out the TD game logic. 3) MQTT publishing 4) Fabrication.

Light System on the ITP Floor

The floor currently has 6 lights in the system, daisy chained to one another (which means the DMX signal goes out from one light into the next); specs about them and further explanations can be found here

Based on a Virtual Console Tom Igoe built and poking around different channels on QLC+’s Simple Desk, I came up with a complete light plot for the DMX system on the ITP floor, with different channels correlating to different functions.

IP Address:

Adding two moving heads from the wonderful Nico Cabalquinto !!

UI Interface

Game Flow in TouchDesigner

I knew it was hard when we sketched out the system diagrams, but this still turned out to be the most challenging, complex system of things I’ve ever made. Since we’re short on time and have A WHOLE LOT on our hands, I decided to tackle the game logic while Nick, the wonderful Python genius that he is, work on the Raspberry Pi algorithm, and then we would try to figure out MQTT communication together.

Here’s a screenshot of the final TouchDesigner sketch:

It’s a bit hard to see, but here’s a basic breakdown.

The top left section is the “main” part of the sketch. It shows a video feed that came from the Kinect, and all the Kinect skeleton data that it picks up; a couple of containers (operators inside shown in the other three sections) for DMX control, game cues, and MQTT communication can also be seen, and they eventually all merged into a switch operator, feeding into a DMX Out CHOP.

Bottom left is the UI interface I built to control the floor light system + 2 moving heads. We didn’t use it in the actual game, but it was very useful to have in the sketch.

Top right is MQTT stuffs with a bit of code. I’ll explain further in the next section.

Bottom right is game logic and ALL THE CUES !!!! It was… a lot. There are four stages of the game: start, pause for pose, dance again, game over + voice line based on final score. Except for the first cue (triggered by button press), the other three are all triggered after certain time passed since last cue. With each stage there are different sound cues lining up, and different lighting behaviors.

We wrote some scripts for the sound cues, and asked our friend Duncan to record them for us. He sounded amazing. Here’s a video of when I first got button trigger to work with Timer CHOP, in order to count the time and auto play a second sound cue (dance music) when first cue (Duncan’s welcome message) ends.

Leave a Reply

Your email address will not be published.