Pages: [1]   Go Down
Author Topic: Interactive Helmet + Glove project. Arduino/Processing interfacing  (Read 1714 times)
0 Members and 1 Guest are viewing this topic.
London
Offline Offline
Newbie
*
Karma: 0
Posts: 6
http://maxenceparache.blogspot.com/
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hello everyone,
I'm doing an interactive design project based on Kinect. The device I'm creating is composed of an helmet of my own design with Headplay video glasses embedded. The kinect captor on the chest (still have to design a kind of box or casing..), then a glove of my own design as well, with 2x force sensing sensors at the extremities of the fingers. The project is meant to be an immersive experience for any user, placing him in the 3D scanned environment (basic 3D point cloud). The glove allows the user to rotate a 3D camera in the live-scanned environment, leaving the hands free of any pad or controler.
Everything is doing quite well but I'll have to be finished at the begining of june and the weak part of my project is the code. I'm a designer, not a programmer, even if I'm interested in creative coding I have a serious lack of knowledge. I wanted to run the kinect hack with OpenFrameworks but didn't manage to do it either. Quite frustrating..
Indeed, the code I'm using is a modified version of Daniel Shiffman with basic camera movement using the PeasyCam library. I also wanted to modify the point of cloud according to environemental sounds but I didn't manage to do it.
My main question is about connecting the data recorded with the sensors on arduino and using it in Processing.
This is the circuit (super basic), recording two values for each finger. On the processing code running kinect I set two keyboard keys to rotate the camera and I just want to replace the keys by the force sensors. I tried to find the good topic, firmata,.. but I didn't manage to have everything working yet.



You can have a look on my research blog (even if it's meant to be for my personal use not for public..). If you're interested in the project reply or send me a message. Any help is more than welcome! Thank you
Logged


Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 644
Posts: 50487
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
My main question is about connecting the data recorded with the sensors on arduino and using it in Processing.
But what is the question? The sensor data can be sent to Processing. In the serialEvent callback, you read the data that the Arduino sent, and either do something directly with it, or store it in global variables that the draw() function (or something that it calls) can use.

Quote
You can have a look on my research blog (even if it's meant to be for my personal use not for public..)
Not unless you provide a link...
Logged

London
Offline Offline
Newbie
*
Karma: 0
Posts: 6
http://maxenceparache.blogspot.com/
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

But what is the question? The sensor data can be sent to Processing. In the serialEvent callback, you read the data that the Arduino sent, and either do something directly with it, or store it in global variables that the draw() function (or something that it calls) can use.

Not unless you provide a link...

- With my actual code I'm sending the values on the Serial monitor but then everything I tried from this was unsuccessful.
The force sensor records values from 0 to 1000. Something like "for 500+, send the value to processing & rotate camera on the 'left' "
I would like to replace the two keyboard keys with the two sensors, activated from a specific value (a real pressure of the thumb on the sensor to leave the user free to manipulate objects without activating the rotation of the camera) and I'd like to know how to do it in the most simple way.

- The blog is linked in my signature smiley-wink http://maxenceparache.blogspot.com/
Logged


Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 644
Posts: 50487
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm confused. Are you trying to make the Arduino send the sensor data to Processing? Or are you trying to make the Arduino use the sensor data directly, instead of sending it to an external application?

Quote
With my actual code I'm sending the values on the Serial monitor but then everything I tried from this was unsuccessful.
You can't use the Serial Monitor and Processing at the same time. The Arduino is on one end of the serial port. Only one application on the PC can be on the other end of the serial port.

Anyway, we can't help you with the code if you won't post it.
Logged

London
Offline Offline
Newbie
*
Karma: 0
Posts: 6
http://maxenceparache.blogspot.com/
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Sorry if I'm not clear. I'm trying to make the Arduino send the sensor data to Processing, yes smiley

Code:
    //FSRx2 test sketch.


    int fsrPin = 0;
    int fsrPin2 = 1;
    int fsrReading;
    int fsrReading2;

    void setup(void) {
    Serial.begin(9600);
    }

    void loop(void) {
    fsrReading = analogRead(0);
    fsrReading2 = analogRead(1);


    Serial.print("Analog reading = ");
    Serial.print(fsrReading);

    if (fsrReading < 10) {
    Serial.println(" - No pressure");
    } else if (fsrReading < 200) {
    Serial.println(" - 200 ");
    } else if (fsrReading < 500) {
    Serial.println(" - 500 ");
    } else if (fsrReading < 800) {
    Serial.println(" - 800");
    } else {
    Serial.println(" - 1000 ");
    }
    delay(1000);

    Serial.print("Analog reading2 = ");
    Serial.print(fsrReading2);

    if (fsrReading2 < 10) {
    Serial.println(" - No pressure");
    } else if (fsrReading2 < 200) {
    Serial.println(" - 200");
    } else if (fsrReading2 < 500) {
    Serial.println(" - 500 ");
    } else if (fsrReading2 < 800) {
    Serial.println(" - 800 ");
    } else {
    Serial.println(" - 1000 ");
    }
    delay(1000);
    }
Logged


Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 644
Posts: 50487
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

EVERYTHING written to the serial port will be available to read in Processing, provided you connect to the correct port, and provide a serialEvent() method to actually read the data.

So, where is your Processing code?
Logged

London
Offline Offline
Newbie
*
Karma: 0
Posts: 6
http://maxenceparache.blogspot.com/
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Here is the Processing code

Code:
import org.openkinect.*;
import org.openkinect.processing.*;
import fullscreen.*;

FullScreen fs;

// Kinect Library object
Kinect kinect;

float a = 0.0f;

int w = 640;
int h = 480;


//lookup table repeat math
float[] depthLookUp = new float[2048];

void setup() {
  size(1440,900,P3D);
  kinect = new Kinect(this);
  kinect.start();
  kinect.enableDepth(true);
fs = new FullScreen(this);
  kinect.processDepthImage(false);

  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
     fs.enter();
  }
}


void keyPressed() {
  if (key == 's') {
  a += -0.03f;
  } else if(key == 'd') {
    a += 0.03f;
  }
   
 }

void draw() {

  background(0);
  fill(255);
  textMode(SCREEN);
  text("Kinect FR: " + (int)kinect.getDepthFPS() + "\nProcessing FR: " + (int)frameRate,10,16);

  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepth();

  // draw every x pixels
  int skip = 2;


  translate(width/2,height/2,-200);
  rotateY(a);

  for(int x=0; x<w; x+=skip) {
    for(int y=0; y<h; y+=skip) {
      int offset = x+y*w;

      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      PVector v = depthToWorld(x,y,rawDepth);

      stroke(255);
      pushMatrix();
      // scale
      float factor = 800;
      translate(v.x*factor,v.y*factor,factor-v.z*factor);
      // Draw a point
      point(0,0);
      popMatrix();
    }
  }




 
}

// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
  if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  depthLookUp[depthValue]; //rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}


void stop() {
  kinect.quit();
  super.stop();
}
Logged


Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 644
Posts: 50487
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

There is nothing in that Processing code that connects to the serial port, and there is no serialEvent() method to read the serial data that is received (of which there currently is none).

Several Processing examples include connecting to the serial port, and reading data from the port. Take a look at one of them to see how it is done.
Logged

London
Offline Offline
Newbie
*
Karma: 0
Posts: 6
http://maxenceparache.blogspot.com/
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Indeed I didn't include any of my unsuccessful lines of code related to my question. This is only the code related to the kinect.
I'll try to find some Processing example as you mentioned
Logged


London
Offline Offline
Newbie
*
Karma: 0
Posts: 6
http://maxenceparache.blogspot.com/
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hello again,

I tried few things with the code but I still don't understand how processing is supposed to read the data recorded by the sensors...
Here is the tutorial I found, which part should I use ?
http://www.learningprocessing.com/examples/chapter-19/example-19-8/

I modified my old code with this tutorial except this part. :

void draw() {
  // The serial data is used to color the background.   
 background(val);
}

This is a screenshot of my arduino code running, and the code follows


ARDUINO :
Code:
/* FSR simple testing sketch.
 
Connect one end of FSR to power, the other end to Analog 0.
Then connect one end of a 10K resistor from Analog 0 to ground
 
For more information see www.ladyada.net/learn/sensors/fsr.html */
 
int fsrPin = 0;
int fsrPin2 = 1;// the FSR and 10K pulldown are connected to a0
int fsrReading;
int fsrReading2;// the analog reading from the FSR resistor divider
 
void setup(void) {
  // We'll send debugging information via the Serial monitor
  Serial.begin(9600);   
}
 
void loop(void) {
  fsrReading = analogRead(0);
  fsrReading2 = analogRead(1);
 
 
  Serial.print("Analog reading = ");
  Serial.print(fsrReading);     // the raw analog reading
 
  // We'll have a few threshholds, qualitatively determined
  if (fsrReading < 10) {
    Serial.println(" - No pressure");
  } else if (fsrReading < 200) {
    Serial.println(" - Light touch");
  } else if (fsrReading < 500) {
    Serial.println(" - Light squeeze");
  } else if (fsrReading < 800) {
    Serial.println(" - Medium squeeze");
  } else {
    Serial.println(" - Big squeeze");
  }
  delay(1000);
 
   Serial.print("Analog reading2 = ");
  Serial.print(fsrReading2);     // the raw analog reading
 
  // We'll have a few threshholds, qualitatively determined
  if (fsrReading2 < 10) {
    Serial.println(" - No pressure");
  } else if (fsrReading2 < 200) {
    Serial.println(" - Light touch");
  } else if (fsrReading2 < 500) {
    Serial.println(" - Light squeeze");
  } else if (fsrReading2 < 800) {
    Serial.println(" - Medium squeeze");
  } else {
    Serial.println(" - Big squeeze");
  }
  delay(1000);
  }
  //if fsrReading & fsrReading2 > 650, send "1" to processing else send "0".
  //for "1" rotate camera of +0.10 in the choosen direction.

PROCESSING:
Code:
import org.openkinect.*;
import org.openkinect.processing.*;
import fullscreen.*;
import processing.serial.*;

FullScreen fs;

int val = 0; // To store data from serial port, used to color background
Serial port; // The serial port object
// Kinect Library object
Kinect kinect;

float a = 0.0f;

int w = 640;
int h = 480;


//lookup table repeat math
float[] depthLookUp = new float[2048];

void setup() {
  size(1440,900,P3D);
  kinect = new Kinect(this);
  kinect.start();
  kinect.enableDepth(true);
fs = new FullScreen(this);
  kinect.processDepthImage(false);

// Using the first available port (might be different on your computer)
  port = new Serial(this, Serial.list()[0], 9600);
 
  // Lookup table for all possible depth values (0 - 2047)
  for (int i = 0; i < depthLookUp.length; i++) {
    depthLookUp[i] = rawDepthToMeters(i);
     fs.enter();
  }
}


void keyPressed() {
  if (key == 's') {
  a += -0.05f;
  } else if(key == 'd') {
    a += 0.05f;
  }
   
 }
 
 void draw() {
  // The serial data is used to color the background.   
 background(val);
}

// Called whenever there is something available to read
void serialEvent(Serial port) {
  // Data from the Serial port is read in serialEvent() using the read() function and assigned to the global variable: val
  val = port.read();
  // For debugging
  // println( "Raw Input:" + val);
}


void draw() {

  background(0);
  fill(255);
  textMode(SCREEN);
  text("Kinect FR: " + (int)kinect.getDepthFPS() + "\nProcessing FR: " + (int)frameRate,10,16);

  // Get the raw depth as array of integers
  int[] depth = kinect.getRawDepth();

  // draw every x pixels
  int skip = 3;


  translate(width/2,height/2,-300);
  rotateY(a);

  for(int x=0; x<w; x+=skip) {
    for(int y=0; y<h; y+=skip) {
      int offset = x+y*w;

      // Convert kinect data to world xyz coordinate
      int rawDepth = depth[offset];
      PVector v = depthToWorld(x,y,rawDepth);

      stroke(255);
      pushMatrix();
      // scale
      float factor = 600;
      translate(v.x*factor,v.y*factor,factor-v.z*factor);
      // Draw a point
      point(0,0);
      popMatrix();
    }
  }




 
}

// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
  if (depthValue < 2047) {
    return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
  }
  return 0.0f;
}

PVector depthToWorld(int x, int y, int depthValue) {

  final double fx_d = 1.0 / 5.9421434211923247e+02;
  final double fy_d = 1.0 / 5.9104053696870778e+02;
  final double cx_d = 3.3930780975300314e+02;
  final double cy_d = 2.4273913761751615e+02;

  PVector result = new PVector();
  double depth =  depthLookUp[depthValue]; //rawDepthToMeters(depthValue);
  result.x = (float)((x - cx_d) * depth * fx_d);
  result.y = (float)((y - cy_d) * depth * fy_d);
  result.z = (float)(depth);
  return result;
}


void stop() {
  kinect.quit();
  super.stop();
}


thank you...
Logged


Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 644
Posts: 50487
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
val = port.read();
reads one byte from the serial port.

Code:
int val = 0; // To store data from serial port, used to color background

What color is the string "Analog reading2 = 600 - Medium squeeze", when read one byte at a time, supposed to correspond to?

I think that you need to break the Processing project up into steps. First, get Processing to echo the data send to the serial port, by the Arduino, one character at a time.

Then, change the Processing application so it reads an entire buffer of data. That requires the use of the bufferUntil() function in setup(), and requires that the Arduino send the specific buffer until character (typically a carriage return). Again, just echo the input.

Then, change the Arduino code to get rid of the extra crap it is sending. Just send the analog values as strings.

Then, the Processing application can be revised to convert the received string to whatever data type is appropriate, int in this case, and store it in the appropriate variable that draw() uses to do something.
Logged

Pages: [1]   Go Up
Jump to: