Programming a robot that learns to walk

Greetings, my fellow people!

I'm currently trying to build a simple, 4 legged, spider-like robot, that can learn by itself, how to walk.
The obvious path would be a Neural Network, but I can quite get a concept of the program structure in my head.

1)Essentially:

  • There are 4 servos, that can turn the legs. They are connected via a PCA9685 and accessed with the "HCPCA9685" library.
  • Mounted on top, sits a "MPU6050" gyro/accelerometer, which gives feedback on forward acceleration.
  • It's all connected to te SDA/SCL pins of an Arduino Uno. The PC sends serial commands to the Arduino via Phyton (pyserial), consisting of the Servo ID and Degree to move a servo.
  • Arduino then returns the acceleration data back to the PC

So, the Neural Network has to be built in Phyton, yet I don't really know how to structure it.
I'm not even sure if a Neural Network is the way to go, as there isn't any data input on which it has to react.

It's more like trial and error learning, where it tries some random moving pattern, checks how far it got, mutates the pattern a bit, checks how far it got and compares it to the previous result, mutates the pattern on that comparison result, etc. etc.

There comes the next problem:
When I have a multiple step movement pattern, how would the program know, wich part of a movement to change and wich changed movement is responsible for better of worse movement?

So more than specific code, I'd be glad for some concept ideas, how would you guys program this?
If you need more information, pictures, code, etc. I can post it later, not at home right now..

Thank you for your help!
Have a nice weekend everybody!
GeriX

It's more like trial and error learning, where it tries some random moving pattern, checks how far it got, mutates the pattern a bit, checks how far it got and compares it to the previous result, mutates the pattern on that comparison result

That is the way I would go.

You need to score success for the genetic algorithm. One method would be to evaluate "walking in a reasonably straight line".

Unfortunately, the MPU6050 is a very poor choice of sensor, as it nearly impossible to use it (or any consumer grade IMU) to estimate position. If you are interested in learning why, see this explanation.

Since you are using a PC, put an IR LED on top of the robot to act as a marker and use a simple overhead USB camera, connected to the PC, to locate it.

GeriX1:
I'm currently trying to build a simple, 4 legged, spider-like robot, that can learn by itself, how to walk.

I'm struggling with the idea that a machine might learn to walk. Surely the ability to walk is built in by the designer?

If not, what is the starting point in terms of what the robot "knows"?

And how will things be different when it knows how to walk. What defines the ability to walk? How will YOU know that it knows how to walk?

Animals are pre-programmed with "how to walk" (let's not bother with a discussion of who wrote that program :slight_smile: )

...R

I'm struggling with the idea that a machine might learn to walk.

The idea is very simple. The machine tries out billions of different ways of moving the legs, and picks the "best" one that moves the machine in a useful way.

In evolutionary terms, the "best" way would be the one that gets to food (energy), escapes destruction (predators) and leads to successful continuation of the germ line. No designer needed.

jremington:
The idea is very simple. Try out billions of different ways of moving the legs, and pick the “best” one that moves the machine in a useful way.

The question is time. You need billions of trials. Each trial take some time to run. How much time do you have? How long do you think the Arduino will last?

I’ve seen this done in silico with really fast computers but there we can do millions of simulations in a matter of milliseconds.

Even if you only let the robot try for 30 seconds at a time and the update is instantaneous so that you can get two trials per minute, it will be almost a thousand years before you have tried a billion times.

Fewer than a billion would probably work.

jremington:
Fewer than a billion would probably work.

How large is the set of random movement possibilities? It's pretty big. If you don't have some way to direct the learning it may take more than a billion.

Point being that it takes more than, "I heard about these things called ANN can I do them with Arduino." You'd need some pretty advance knowledge about ANN and how they work and how to train them. The kind of knowledge that the OP seems to demonstrate a lack of. It's one of those, "if you have to ask then the answer is probably no" type of situations. The people who could do this would know at least in a general sense how they would go about getting started and wouldn't be asking us.

Well there you go. Seems they took the approach of limiting that search space. So this one only has three possible positions for each leg. So it's not going to be a very pretty "walk" but they claim it did learn.

I was thinking of a more complicated bot that was actually "walking" in the traditional sense. I guess if you limit what you consider walking to simply moving forward over a smooth surface then you can reduce this problem quite a bit.

The Hackaday page and original project description is a really nice find, thanks!

Something to slaver over Introducing Spot Classic (previously Spot) - YouTube

...R

That is the $100M version, courtesy of the obscenely bloated U.S. defense department budget.

It works also with human models Google's DeepMind AI Just Taught Itself To Walk.

So!
As already suggested, I tried to limit the possible combinations as best as possible.
The idea was to only make every 'Generation' of movement pattern about 5 seconds long and see how far it gets.
As with straight forward movement, if you have a working move figured out, just loop it and it'll go forever.

A moving pattern would have to movements per servo max.

Maybe look something like this:

  1. Servo 1: 50°
  2. Servo 2: 40°
  3. Servo 3: 120°
  4. Servo 4: 70° The order of the commands and the given values are random in the beginning,
  5. Servo 2: 80° and should be altered through trial and error.
  6. Servo 1: 20°
  7. Servo 3: 85°
  8. Servo 4: 170°

Robin2:
I'm struggling with the idea that a machine might learn to walk. Surely the ability to walk is built in by the designer?

If not, what is the starting point in terms of what the robot "knows"?

The robot knows that it has walked forward, because it gets rewarded under certain circumstances.
To say it simple, a certain change of the moving pattern for forward walking, is good when the measured distance it has travelled is greater than the try before.

jremington:
Unfortunately, the MPU6050 is a very poor choice of sensor, as it nearly impossible to use it (or any consumer grade IMU) to estimate position. If you are interested in learning why, see this explanation.

Since you are using a PC, put an IR LED on top of the robot to act as a marker and use a simple overhead USB camera, connected to the PC, to locate it.

Well, in chosing the MPU, I hoped to be able to use the sum of forward acceleration during the given testing time, as a good feedback value. There isnt really a big amount of accuracy and absolute values nescessary, when it's just important, if one measurement is bigger than another.
I dont have to calculate at wich speed it walks, nor how far it went, just and indication if one pattern is better than another.

Delta_G:
Train Your Robot To Walk With A Neural Network | Hackaday

Well there you go. Seems they took the approach of limiting that search space. So this one only has three possible positions for each leg. So it's not going to be a very pretty "walk" but they claim it did learn.

I was thinking of a more complicated bot that was actually "walking" in the traditional sense. I guess if you limit what you consider walking to simply moving forward over a smooth surface then you can reduce this problem quite a bit.

Thats a good find indeed, shall look into it and see how I could translate it to my project, thanks!

Delta_G:
Point being that it takes more than, "I heard about these things called ANN can I do them with Arduino." You'd need some pretty advance knowledge about ANN and how they work and how to train them. The kind of knowledge that the OP seems to demonstrate a lack of. It's one of those, "if you have to ask then the answer is probably no" type of situations. The people who could do this would know at least in a general sense how they would go about getting started and wouldn't be asking us.

Well yes, thats one way to see it. I may not have a degree in programming, yet I have interest and motivation to learn more, than I am taught in my current field of work. I already did learn more than I have ever thought with arduino and am not scared to face more dificult topics. Furthermore, I wasn't asking for someone to give me a complete lecture on ANN, but rather if someone already has experience or even an idea on how to solve this program. Thats what a community is for after all I'd suggest.

Delta_G:
Train Your Robot To Walk With A Neural Network | Hackaday

Well there you go. Seems they took the approach of limiting that search space. So this one only has three possible positions for each leg. So it's not going to be a very pretty "walk" but they claim it did learn.

I was thinking of a more complicated bot that was actually "walking" in the traditional sense. I guess if you limit what you consider walking to simply moving forward over a smooth surface then you can reduce this problem quite a bit.

Well, in chosing the MPU, I hoped to be able to use the sum of forward acceleration

Back and forth lurches in place will have a large sum of forward accelerations. Using an accelerometer will be an excellent learning experience.

Well yes, thats one way to see it. I may not have a degree in programming, yet I have interest and motivation to learn more, than I am taught in my current field of work. I already did learn more than I have ever thought with arduino and am not scared to face more dificult topics. Furthermore, I wasn’t asking for someone to give me a complete lecture on ANN, but rather if someone already has experience or even an idea on how to solve this program. Thats what a community is for after all I’d suggest.

I think you missed my point. My point wasn’t that you can’t. My point was that you might do well to learn a bit about ANN before you jump into the deep end. I literally meant to learn to crawl before you learn to walk. You simply cannot do this if you don’t have an understanding of ANN and this isn’t a very good way to gain that. Start with something where you can learn the basic topologies of ANN systems and learn how they work. Start from the beginning. You’ll have a much higher chance of success if you do.

The Wright brothers didn’t start out by pointing their airplane out to sea. They didn’t fancy their first flight to cross the atlantic. They just tried to make it from one sand dune to the next.

As noted, not only does your DNA contain the instructions to build the body, but it also pre-wires the brain with directions to run the body.

Animals do not just figure out how to walk, but learn how to walk according to their heritage.

Even if DNA only half programmed a animal, a horse might learn to “walk” by jumping, hopping, two legs, three legs, 4 legged skip, and tons of other possible ways to move with 4 legs...

I always remember my physics teacher showing us how walking is nothing more that a “controlled fall”.

Slumpert:
Even if DNA only half programmed a animal, a horse might learn to “walk” by jumping, hopping, two legs, three legs, 4 legged skip, and tons of other possible ways to move with 4 legs...

But the reality is that it does not. A foal can walk within a few minutes of being born. In the wild, if he couldn't, he was eaten.

I can understand the interest in machine learning. I just don't think learning to walk is an effective experiment for this. If you start from the position of the machine having zero knowledge then the task is too big. And if you start with 50% knowledge of how to walk then what exactly do you put into that 50% and what is the machine actually learning?

...R

Robin2:
I can understand the interest in machine learning. I just don't think learning to walk is an effective experiment for this. If you start from the position of the machine having zero knowledge then the task is too big. And if you start with 50% knowledge of how to walk then what exactly do you put into that 50% and what is the machine actually learning?

...R

Well the whole point of the robot was to experiment with this technology and to get an idea on how to develop it. So either it can learn from 0 on, or you could as well not start with random starting values, but with values programmed in by you, that already resemble walking. The machine would then optimize itself and learn to walk faster and better.

GeriX1:
I’d be glad for some concept ideas, how would you guys program this?

Here’s my suggestion fwiw.

Download vd Smagt and Krose’s An Introduction to Neural Networks. Get your mind round Fig 2.1 and then code a learning Perceptron ala Fig 3.2 and Fig 3.3.

Take it from there: get a few layers of Perceptrons going.

Without that kind of personal learning about the concepts of neural nets, I don’t think you should consider an actual hardware project. (Of course, nothing wrong with keeping some kind of end goal in mind as you develop a basic understanding of the techniques.)

You can do a huge amount of learning about how ANN’s work without going anywhere near a robot; indeed I think it’s essential to build up a library of knowledge first. (And by library I mean knowledge in your own mind, as well as a library of software tools you can call on when you move on to a hardware problem.)

Don’t forget that even as we speak, loads of folk are doing their PhDs and are in post-doc fellowships working on stuff like this. So yeah vd Smagt and Krose’s Intro is over 20y old, but I think you need to prove to yourself you can code a Perceptron and then stick a few together in layers.

arduin_ologist:
Here's my suggestion fwiw.

Download vd Smagt and Krose's An Introduction to Neural Networks. Get your mind round Fig 2.1 and then code a learning Perceptron ala Fig 3.2 and Fig 3.3.

Take it from there: get a few layers of Perceptrons going.

Without that kind of personal learning about the concepts of neural nets, I don't think you should consider an actual hardware project. (Of course, nothing wrong with keeping some kind of end goal in mind as you develop a basic understanding of the techniques.)

You can do a huge amount of learning about how ANN's work without going anywhere near a robot; indeed I think it's essential to build up a library of knowledge first. (And by library I mean knowledge in your own mind, as well as a library of software tools you can call on when you move on to a hardware problem.)

Don't forget that even as we speak, loads of folk are doing their PhDs and are in post-doc fellowships working on stuff like this. So yeah vd Smagt and Krose's Intro is over 20y old, but I think you need to prove to yourself you can code a Perceptron and then stick a few together in layers.

+1000000

Don't discourage yourself by jumping in the deep end before you learn to swim. This is going to be a really frustrating and tough thing if you don't already understand the basics of ANN. There are some basic ANN projects (like the perceptron) that you can build with just a few neurons so you can learn the basics and how to control them. Once you spend some time with that, then the walking project will make a LOT more sense.