Simple MLP - NeuralNetwork Library

Projects Showcase
1

Hello people,

[ A Breif introduction of myself ]

I am George, i am yet a non-professional programmer that loves coding and computers since i was a kid, started programming from the age of as young as 14-15 years old.

[ About The Project [...] ]

I've made a NeuralNetwork Library as you probably realised from the Headline, that it is realativly simple to use, (thanks to you and your help in this forum). Please have in mind that i did my best to make it as better as i could and that yet i am not a professional programmer. (I have achived running a sketch of a Neural-Network even on an attiny85)

[ In Conclusion ]

Please feel free to ask me anything and tell me any of your thoughts or your pros and cons about my project. With love and respect, i wish you the happiest day people[...]

Github's Link to the Library

PS. I need some motivation please [...]

[ NEW VERSION UPDATE 30/1/2021 ]
go check it out people :smiley:

2

What sort of project might somebody use your Arduino neural network for?

...R

3

Thanks for your contribution to the Arduino community Giorgos_xou!

I had a quick look over the repository for common library problems and didn't find a single one. Usually I can find at least one problem with any given library.

I saw that you added a "Hastags" section to make it easier for people to discover your library via keyword searches. Towards that goal, you might consider using GitHub's excellent Topics system:

If you want to provide the users of your library with an easy way to install and update, you might consider adding the library to the Arduino Library Manager index:

4

Robin2:
What sort of project might somebody use your Arduino neural network for?

Such a nice question! (:

in just two words i could say, almost for any "smart-thing";

but lets say for example, that you want to have a vacuum cleaner robot, that will be able to adapte in the enviroment based on its sensor's inputs. For this kind of robot most of the people would use a Neural-Network [...]

Another example of implementing MLP-NNs could be for making a smart-home, that based on user's feedback or whatever, lets say that it starts learning how to adapt the lightings or the music volume based on where the user is located inside the home [...]

In my case, i've started making this library about some months ago, for a college project i haven't done yet (and not only)... that it has to do with using a low resolution camera with an ATtiny microcontroller and achieving the recognition of letters and numbers with the help of a pretrained Neural-Network.

i say adapt and i mean "backpropagation" ,something that it is quite computationally heavy for Arduino UNO like microcontrollers (that's why there is also an option for loading pretrained NNs or even using PROGMEM if your microcontroler doesn't have enough SRAM, in my Library)

Thanks you for asking (:

Robin2:
...R

R is a programming language and free software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing [...] ? or just Robin?

5

pert:
Thanks for your contribution to the Arduino community Giorgos_xou!

It is my pleasure!

pert:
I had a quick look over the repository for common library problems and didn't find a single one. Usually I can find at least one problem with any given library.

Thanks you for taking time to look even just briefly my repository, makes me really happy (: and even more, reading that you havent found any problem ((: [...]

pert:
I saw that you added a "Hastags" section to make it easier for people to discover your library via keyword searches. Towards that goal, you might consider using GitHub's excellent Topics system ...

keyword searches Check :heavy_check_mark:!

pert:
If you want to provide the users of your library with an easy way to install and update, you might consider adding the library to the Arduino Library Manager index ...

... Library Manager Check :heavy_check_mark:!

Also Thanks you for Helping me and for replying in my post (:

6

Giorgos_xou:
in just two words i could say, almost for any "smart-thing";

but lets say for example, that you want to have a vacuum cleaner robot, that will be able to adapte in the enviroment based on its sensor's inputs. For this kind of robot most of the people would use a Neural-Network [...]

Sorry for the late follow-up.

I guess I was not asking about neural networks in general but rather about the sort of project you could implement with a neural network on an Attiny85? Or even on an Atmega 328?

I have always assumed that a neural network would require a great deal more memory and processing power.

...R

7

Robin2:
Sorry for the late follow-up.

Don't worry, it's always fine (:

Robin2:
I have always assumed that a neural network would require a great deal more memory and processing power.

Reason:
And yep! you are right, most of the neural networks require much more memory and especially processing power of that power, an attiny or even an atmega has to give out (and that's why people tend to use Rasbery pi as an example, insted of simple microcontrollers or even more powerfull computers. [something i will mention in a little, after]). But! that comes/happens when the NN needs to-adapt/to-learn/backpropagate, then it is the point where it needs more memory and processing power!(most of the times) not when it has to feedforward/to-execute/to-get-an-input-and-give-an-output.

Explanation:
Very briefly, the reason why (in a briefly [not exactly mathimatical] way) Backpropagation is such a power consuming process, because it needs to find the error and then distribute/share "respectfully" the error to each neuron of each layer, with "respect" to each weight (and the bias) of each front neuron it is connected with [...] !

Where in the case of FeedForward process, it "only" needs to get the inputs or in other words the outputs of the previous layer and then pass them using a simple formula and an activation function (that also plays an important role) from the current layer to the next and then go on until the last layer, the final output-layer.

More info Here

Robin2:
I guess I was not asking about neural networks in general but rather about the sort of project you could implement with a neural network on an Attiny85? Or even on an Atmega 328?

In my example of "vacuum cleaner robot" it would not be the best idea to use an attiny or an atmega, because you need something that adapts alot to the enviromnt unless you have (as i like to call it) an "e-soul" a computer where the lerning happens there and then passes the trained NN to Microcontroller via BLE or whatever, thats why it is preferable to use a Rasbery for those examples or etc.

As for the other examples i gave you, are possible!(at least as i think about them) like in the example of a smart home will also be present a main external-brain/"e-soul" except of the small-brains of each microcontroller [...]

As for the last one is an-example/something i am working on and goes very well till now. and i am very confident that it will work as i have to do with a camera of 225 pixels and an Attiny167

In Sum:
As long as you have the needed amount of SRAM or PROGMEM for the size of the NN, you can do almost anything, the only limitation is the training part i would say, (of course also the specs of the microcontroller) that it is only suitable for high cpu speeds, but for simple and practical examples like the training part of a decision between some simple things like in my example files in library, would be fine, or for something that timing is not something that crucial.

End:
I wish i've made it a little bit clearer because it is true that i wasn't that clear with what i was saying in a way.
And also please people, be mindfull of the fact that i am not yet a professional programmer, so i might have explained some things not in the proper way (: [...]

Thanks for reading,
George

8

Giorgos_xou:
As for the other examples i gave you, are possible!(at least as i think about them) like in the example of a smart home will also be present a main external-brain/"e-soul" except of the small-brains of each microcontroller [...]

As for the last one is an-example/something i am working on and goes very well till now. and i am very confident that it will work as i have to do with a camera of 225 pixels and an Attiny167

Thank for the additional explanation.

I know very little about neural networks but, if something interesting could be done with an Arduino I may take the trouble to explore it.

I think I understand the distinction you are making between the Back Propagation and the Feed Forward activities - that sounds like the "learning" and "doing" phases to me. However I have always assumed the essential part of a neural network is its ability to learn. How can the Feed Forward part work if the system has no knowledge?

I must confess that your reference to the smart home is not specific enough for me to envisage how an Arduino based neural network might be used.

It would be interesting to hear more about what you can do with the low-res camera - in particular what can be done with a neural network more easily than with a conventional program.

...R

9

Robin2:
Thank for the additional explanation.

You're welcome. (:

Robin2:
I think I understand the distinction you are making between the Back Propagation and the Feed Forward activities - that sounds like the "learning" and "doing" phases to me

Exactly! Back-Propagation is the "Learning" process and Feed Forward is the "doing"/"run"/"execution" process of the NeuralNetwork; the part that you give some inputs and the NeuralNetwork tells you what it gussed [...]

Robin2:
However I have always assumed the essential part of a neural network is its ability to learn. How can the Feed Forward part work if the system has no knowledge?

It won't be "clueless" of what output-to-give/decicion-to-take because it will be already pre-trained, so that it won't need to learn anything else [...]

In the same concept/way you are pre-trained for an upcoming race/contest because you can't always learn-fast and adapt to the needs/conditions of the race/contest instantly or in the compulsory time, you might hear the teachings of a master insted of trying to learn by yourselft about the tricks and ways, so you can have a greater chances to win the contest, a NuralNetwork can be pre-trained too! ready to be "executed"/"done"/"ran" , pretrained by a powerfull-computer-master way faster than from trying by itself as a "simple" microcontroller to train itself.

Robin2:
I must confess that your reference to the smart home is not specific enough for me to envisage how an Arduino based neural network might be used.

I am sorry /:, you have a point here in a way. Sometimes i get a lil-bit crazy with the explenations and i might say more like conceptual things or get into account things as already known by the others [...] (But i think that in the next paragraph i will be clear enough (: )

Robin2:
It would be interesting to hear more about what you can do with the low-res camera - in particular what can be done with a neural network more easily than with a conventional program.

To give you an idea, i have a camera from an optic mouse (called ADNS2610) that it is black and white and a specific lens (in this case the mouse's lens) where i can see some text on a paper, letters or numbers and symbols. To recognise those letters, numbers and symbols using a conventional program it would be a slightly dificult process and most probably huge in size i would say.

So in this case, to make the process way easier and much more smaller in size, you can have a Neural-Network; witch will have (15*15 = 225 [pixels of the camera]) 255 Input neurons in the first layer, some hidden neurons in the hiden layers (based on how many things you want it to recognise) and at the end an output-layer with a number of output-neurons each corresponding to a letter,number or symbol you want to recognise as.

Then, because Attiny or Atmega are way to slow compared to the actual requirements for training the Neural Network, we decide to use an external computer witch will do our "heavy" work; training way to faster than the time it would take for the microcontroler to train the Neural Network, based on images you took before using the camera or based on images you take the exact same time you run the program/sketch and then at the end passed them to the computer via BLE or a Serial port or etc. (to do the training)

In Sum:
So in a way it is like: Gathering Information via the camera > passing Information to the extrernal Computer via ble or Serial or etc> Training the Neural Network in the external Computer > passing the now pre-trained Neural Network (from the external Computer) to the microcontroller (witch in fact you just pass some Weight-values between the connection of each neuron and not the actual NeuralNet [...]) > and finaly it is ready to run/be-executed with or even without the need of an external computer (because now it knows how to act, what outputs to expect based on it's [enviroment] inputs ).

Thanks for your interest and thanks For Reading,
George. (:

[...]

10

Thank you again.

I think I will leave it at that. I can't say that I am any closer to understanding how I might usefully use an Arduino based neural program.

...R

11

Well done Giorgos_xou!
I thought to test it on STM32 MCU, a cheap bluepill has 20K RAM and 64K flash.
Suggestion: use uint16_t instead of unsigned int because on STM32 it will be a uint32_t

@Robin2
Think at neural networks as a universal interpolator that can be computed fast.
I remember that was used by Samsung on their fridges to learn how to optimize the fridge cycle upon user open close frequency along the day.

12

@Robin2

Robin2:
Thank you again.

You are welcome but thanks you too; for reading and for showing your interest for my project (:

Robin2:
I think I will leave it at that. I can't say that I am any closer to understanding how I might usefully use an Arduino based neural program.

That's ok, however i highly recomend to check out just only the first video of this playlist if you would love to know more in general, it is by far the best playlist i would say.

@zoomx

zoomx:
Well done Giorgos_xou!

Thanks you very much! (:

zoomx:
I thought to test it on STM32 MCU, a cheap bluepill has 20K RAM and 64K flash.

Have you tested it yet? and did it work?

zoomx:
Suggestion: use uint16_t instead of unsigned int because on STM32 it will be a uint32_t

Just by curiosity, does it work on the STM32 if it is unsigned int or not? I think that it would be better if i had #defined-directive-property for selecting between uint8_t .. uint64_t and maybe as default uint32_t . Also should i use uintx_t for the For-loops too? Thanks for the suggestion!

zoomx:
I remember that was used by Samsung on their fridges to learn how to optimize the fridge cycle upon user open close frequency along the day.

Thats very intresting and a very nice example! +1

BTW Does anyone know if STM32 supports double precision? or double precision is only supported by 64bit microcontrollers?

13

I just tested only a compilation using the official ST core and the STM32duino core and I got no errors for Backpropagation_double_Xor example. The same for the ESP8266 and ESP32 cores.

There are some ESP32 boards that have a small cam usually an OV2640 sensor, so a neural network can be useful.

STM32 is a big ARM MCU family, some of them has FPU unit. Double occupies 64-bit (8-bytes). It is the same for ESP8266 and ESP32, they are 32bit MCU. Usually only 8bit MCU have float and double that occupies 4 bytes.

Unfortunately I don't have the hardware near me to load and test but I don't expect errors.

14

zoomx:
I just tested only a compilation using the official ST core and the STM32duino core and I got no errors for Backpropagation_double_Xor example. The same for the ESP8266 and ESP32 cores.

Nice!

zoomx:
There are some ESP32 boards that have a small cam usually an OV2640 sensor, so a neural network can be useful.

Intresting..

zoomx:
STM32 is a big ARM MCU family, some of them has FPU unit. Double occupies 64-bit (8-bytes). It is the same for ESP8266 and ESP32, they are 32bit MCU. Usually only 8bit MCU have float and double that occupies 4 bytes.

Now i see how i can fix that in my library, Thanks! (:

15

Just tested on a Maple Mini STM32F103 at 72MHZ, Backpropagation_double_Xor example.
About 151 seconds to train the network.

On ESP8266 at 80MHz I added some yeld(); here

      NN.FeedForward(inputs[j]); // Feeds-Forward the inputs to the first layer of the NN and Gets the output.
      yield();
      NN.BackProp(expectedOutput[j]); // Tells to the NN if the output was right/the-expectedOutput and then, teaches it.
      yield();

because without the chip reset since the same core had to do other tasks. Maybe I have to disable WiFi.
It took about 131 seconds

16

zoomx:
Just tested on a Maple Mini STM32F103 at 72MHZ, Backpropagation_double_Xor example.

Nice! i will add it in the "tested" section [...] (with a reference-link [tested by] to your profile here? if it is fine/ok by you)

zoomx:
On ESP8266 at 80MHz I added some yeld(); here

Fisrt time i hear about yeld(); i didn't know what it was, I searched about it and it is pretty interesting..

zoomx:
It took about 131 seconds

I am sure that can be way faster, If i wasn't deleting and recreating some arrays for SRAM-economy (i just have to have in my mind, to set a property for this too, in the future) but still, 2 minutes doesn't seem that fine for 3000 loops at 80MHz, maybe yeld plays a role too, because if i am not wrong arduino was around 2 minutes too , hmm...but still Thumps-Up for testing! (:

17

Giorgos_xou:
(with a reference-link [tested by] to your profile here? if it is fine/ok by you)

Yessss :slight_smile:
Same sketch on Arduino took about 803 seconds, more that 13 minutes. Loops are 8000!

18

zoomx:
Yessss :slight_smile:

(:

zoomx:
Same sketch on Arduino took about 803 seconds, more that 13 minutes.

!!! O:

zoomx:
Loops are 8000!

You are right! i thought i had 3000 but 3000 was in the other example...

if you play with the learning rates, changing them to a higher number you can have ~3000 loops and still get the ~same outputs.

I should have change the learning rates for this sketch, because it is actually quite slow (and people might think that it doesn't work)

19

Maybe you can use a variable learning rate, higher when start and going down when learning increase.

20

zoomx:
Maybe you can use a variable learning rate, higher when start and going down when learning increase.

True, however i 'll see what i will do until the next version-update..
( Thanks you for helping me out and for your interest, means a lot to me (: )