Dance to the rhythm using your own webcam.

For my final project I am combining this with my Music Interaction Design final project.

In this rhythm game, PoseNet is utilized to hit repetitive beats while playing a drum kit. Players use their bodies as a controller to hit upcoming shapes laid out on the live video capture screen on beat with their wrists. The score is computed by how well they time their dance to the beats being presented. The players explore the relationship between the movements of their bodies, sound, and rhythm. Augmented reality with a webcam creates an accessible way to explore all of these. The project is the start of a game of which has hopes of having more song choices, various levels, and more intricate functionality. 

My first step was to create the interface with sounds. Then with PoseNet using the Left and Right Wrist.

https://editor.p5js.org/natayie/present/E_IQ4p14O

I then coded the Arduino LED lights according to each side and when the side is 'hit'. I came to initial problems with the p5 code as the outByte was sending too much information too fast, and I solved it with frameCount.

int ledPinDown = 2;
int ledPinRight = 3;
int ledPinLeft = 4;
int ledPinUp = 5;

void setup() {
  pinMode(ledPinDown, OUTPUT);  // sets the pin as output
  pinMode(ledPinRight, OUTPUT);
  pinMode(ledPinLeft, OUTPUT);
  pinMode(ledPinUp, OUTPUT);
  Serial.begin(9600);        // initialize serial communications
}
 
void loop() {
 if (Serial.available() > 0) { // if there's serial data available
   int inByte = Serial.read();   // read it
   if (inByte == 1) {
     digitalWrite(ledPinDown, HIGH);  // use it to turn on the LED DOWN
     digitalWrite(ledPinRight, LOW);
     digitalWrite(ledPinLeft, LOW);
     digitalWrite(ledPinUp, LOW);
   } else if (inByte == 2) {
     digitalWrite(ledPinRight, HIGH);  // use it to turn on the LED RIGHT
     digitalWrite(ledPinDown, LOW);
     digitalWrite(ledPinLeft, LOW);
     digitalWrite(ledPinUp, LOW);
   } else if (inByte == 3) {
     digitalWrite(ledPinLeft, HIGH);  // use it to turn on the LED LEFT
     digitalWrite(ledPinDown, LOW);
     digitalWrite(ledPinRight, LOW);
     digitalWrite(ledPinUp, LOW);
   } else if (inByte == 4) {
     digitalWrite(ledPinUp, HIGH); // use it to turn on the LED UP
     digitalWrite(ledPinRight, LOW);  
     digitalWrite(ledPinDown, LOW);
     digitalWrite(ledPinLeft, LOW);
   } else {
     digitalWrite(ledPinUp, LOW); // sets all LEDs OFF
     digitalWrite(ledPinRight, LOW);  
     digitalWrite(ledPinDown, LOW);
     digitalWrite(ledPinLeft, LOW);
   }
   delay(400);                // waits
 }
}

Next Steps:

• Finish the rhythm game
• Connect to larger, tube LEDs
• Stylize interface