Trying to make a hand gesture sensor with complex gestures.

Hey there everyone :slight_smile:

I have a question for a project of mine; I've seen many videos on hand gesture recognition which feature infrared sensors or ultrasonic sensors or both in some but all of them were for just simple gesture recognition. Mostly swipe gestures.

I was wondering if it's possible to do complex gestures like pinching, drawing a circle or anything like that. Since I've never used an ultrasonic sensor, if it's really precise as hell, I guess it might be able to recognise a circular motion of a finger via 2 ultrasonic sensors to generate some form an axis. Am I on the wrong page here?

For instance, does the ultrasonic sensor measure spherically or just directly? If it's directly, then I believe it would not be possible, but spherically... not sure. Infrared is another option but how? Can I generate the so-called axis for example?

I am also aware of something called the ZX Distance and Gesture sensor which sort of generates a 2 axis field which is great for what I'm trying to do but unfortunately that only works to 20-30 cm which is really short for me.

Could you guys please guide me on something like this? Can you point me to some guides/tutorials about distance measuring?

No chance that an ultrasonic sensor could recognize gestures except, perhaps for arm waving. You can measure distance only, to about +/- 1 cm accuracy, over a fairly broad "cone of vision".

Some progress has been made in gesture recognition using 3-axis accelerometers attached to the hand, and with an absolute orientation sensor, you could perhaps do better. Google will tell you more about both.

What kind of ultrasonic sensor are you talking about? If ultrasonic sensors can detect heartbeats of unborn babies still inside their mothers, and calculate body fat percentages inside people, there may be something that could detect hand and finger gestures.

I think I have seen Threads on the Forum involving gloves that have resistors in them whose resistance varies as the fingers are curved. Maybe something like that would work?

...R

Jimmus:
What kind of ultrasonic sensor are you talking about? If ultrasonic sensors can detect heartbeats of unborn babies still inside their mothers, and calculate body fat percentages inside people, there may be something that could detect hand and finger gestures.

Medical ultrasound imaging is very high frequency (> 1MHz), high cost, and operates only in tissue or aqueous gel.
Heart monitor ultrasound is Doppler.

I'm going to place a bet that the OP doesn't have a medical budget, and wants to operate in air, not tissue, and wants to use hobby sensors, which are hard to read at update rate greater than about 25 Hz, and if you've got more than one, then the per-sensor update rate drops off even quicker (e.g. three sensors, 8Hz).
They also have relatively narrow beams, making broad gestures hard to "see" close up, and you have to be close, because their range isn't great.

Thanks for all the answers! :slight_smile:

It's certainly not a medical sensor. :slight_smile:

I ended up using two HC-SR04 sensors with the NewPing library in order to achieve a form of a swipe gesture effect to trigger 2 way 5V relay's INs. Decided that was enough for the project so no longer need complex hand gesture recognition for now. It works well over 150cm and the gesture is perfectly recognised.

Thanks again for the efforts :slight_smile:

That sounds interesting. I can imagine that other people might find other uses for the idea.

Can you tell us more about your final project?

What sort of gestures can it distinguish?
Maybe you could provide a diagram that illustrates how it looks and how it is used?

...R

This might be of passing interest: Google: Project Soli

...even more interesting when you can get those devices at a reasonable price :slight_smile:

Yours,
TonyWilk

Robin2:
That sounds interesting. I can imagine that other people might find other uses for the idea.

Can you tell us more about your final project?

What sort of gestures can it distinguish?
Maybe you could provide a diagram that illustrates how it looks and how it is used?

...R

Cerainly! :slight_smile: I already have it's fritzing diagram here;


It can recognise an "up" swipe and a "down" swipe from approx. 150cm but the maximum for hc-sr04 is way higher up to 400cm which is not neede in my case. I am using this to control a window blind set which works with a remote controller on a motor. A type of step motor but works much differently from common types :slight_smile:

And the code I used is here;

#include <neotimer.h>
#include <LCD.h>
#include <LiquidCrystal_I2C.h>
#include <NewPing.h>
 
#define SONAR_NUM     2 // Number or sensors.
#define MAX_DISTANCE 90// Max distance in cm.
#define PING_INTERVAL 33 // Milliseconds between pings.
#define ledUp 12
#define ledDown 7
 
unsigned long pingTimer[SONAR_NUM]; // When each pings.
unsigned int cm[SONAR_NUM]; // Store ping distances.
uint8_t currentSensor = 0; // Which sensor is active.
uint8_t lastSensor = 0; // Which sensor is active.
int value1;
int value2;

LiquidCrystal_I2C lcd(0x27,2,1,0,4,5,6,7); // 0x27 is the I2C bus address for an unmodified module
Neotimer thetimer = Neotimer(1000);
 
NewPing sonar[SONAR_NUM] = { // Sensor object array.
  NewPing(9, 10, MAX_DISTANCE),
  NewPing(11, 8, MAX_DISTANCE),
};
 
void setup() {

pinMode(ledUp, OUTPUT);
pinMode(ledDown, OUTPUT);
    digitalWrite(ledUp, HIGH);
    digitalWrite(ledDown, HIGH);
  Serial.begin(115200);
  lcd.setBacklightPin(3,POSITIVE);
  lcd.setBacklight(LOW); 
  lcd.begin(16, 2);
  lcd.clear();
  pingTimer[0] = millis() + 75; // First ping start in ms.
  thetimer.start();
  for (uint8_t i = 1; i < SONAR_NUM; i++)
    pingTimer[i] = pingTimer[i - 1] + PING_INTERVAL;
}
 
void loop() {
  for (uint8_t i = 0; i < SONAR_NUM; i++) {
    if (millis() >= pingTimer[i]) {
      pingTimer[i] += PING_INTERVAL * SONAR_NUM;
      if (i == 0 && currentSensor == SONAR_NUM - 1)
      sonar[currentSensor].timer_stop();
      currentSensor = i;
      cm[currentSensor] = 0;
      sonar[currentSensor].ping_timer(echoCheck);
    }
  }
  
  if(thetimer.repeat(300)){
    lcd.setCursor(0,1);
    lcd.print("Reset");
    value1 = 0;
    value2 = 0;
  }
  
    lcd.setCursor(0,0);
    lcd.print("Working");
    lcd.setCursor(0,1);
    lcd.print("No Action");
    
  if(cm[0] > 0 || cm[1] > 0)
  {
    Serial.print("0");
    Serial.print("=");
    Serial.print(cm[0]);
    Serial.print("cm ");
    Serial.print("1");
    Serial.print("=");
    Serial.print(cm[1]);
    Serial.print("cm ");
    Serial.println();
  }
  if (cm[0] > 10){
    value1 = 1;
    thetimer.reset();
    delay(50);
  }
    
  if (cm[1] > 10 && value1 == 1){
    value1 = 0;
    goUp();
    digitalWrite(ledUp, LOW);
    delay(400);
    digitalWrite(ledUp, HIGH);
    lcd.clear();
    value1 = 0;
    value2 = 0;
  }

  //set value 2 true
  if (cm[1] > 10){
    value2 = 1;
    thetimer.reset();
    delay(50);
  }
  
  if (cm[0] > 10 && value2 == 1){
    value2 = 0;
    goDown();
    digitalWrite(ledDown, LOW);
    delay(400);
    digitalWrite(ledDown, HIGH);
    lcd.clear();
    value1 = 0;
    value2 = 0;
  }
}
 void goUp()
 {
   Serial.println("MovingUpStarted");
    lcd.setCursor(0,1);
    lcd.print("Go Up Trigger");
    value1 = 0;
    value2 = 0;
 }
 void goDown()
 {
   Serial.println("MovingDownStarted");
    lcd.setCursor(0,1);
    lcd.print("Go Down Trigger");
    value1 = 0;
    value2 = 0;
 }
void echoCheck() { // If ping echo, set distance to array.
  if (sonar[currentSensor].check_timer())
    cm[currentSensor] = sonar[currentSensor].ping_result / US_ROUNDTRIP_CM;
}

Thanks for sharing. I will bookmark this.

How are the sensors located - how far apart are they?

It may be interesting to explore what could be achieved with 3 of them.

...R

Robin2:
Thanks for sharing. I will bookmark this.

How are the sensors located - how far apart are they?

It may be interesting to explore what could be achieved with 3 of them.

...R

Well they are currently about 20cm apart from each other. I also started a prototype with 4 of them in order to accept 4 gestures like left-right-up-down. But the code became much more complex for that so I just dropped it but the idea is out there :slight_smile:

Also, circular motion is possible to be caught by getting 2 median points in order to determine a sort of XY axis so theoratically circular motion is also possible to be recognised but way harder to code. I just did not have the time yet but will definitely try it soon...

I uplaoded this to github btw :slight_smile: