Optimization for low memory

Hi,

I wrote an ‘Auth’ library which mainly crates a 332 bytes password for MQTT connection. By now I’ve consumed 89% of flash and 71% of SRAM.

Sketch uses 28,786 bytes (89%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,457 bytes (71%) of dynamic memory, leaving 591 bytes for local variables. Maximum is 2,048 bytes.

Attachment mqtt_test.ino mainly has 3 parts.

  1. Init Ethernet
  2. Generate password
  3. Connect to MQTT and send message

At the top of the sketch I have 3 macros for testing. Enable ETHERNET_TEST and AUTH generates right password. Enable ETHERNET_TEST and MQTT_TEST can connect to MQTT and send message successfully. However, enable ETHERNET_TEST, MQTT_TEST, and AUTH cannot work properly.

The system will hang at the stage after initializing Ethernet.
Log:

memory(start): 
519
Initializing Ethernet...Connected with IP: 192.128.1.151
memory: 
419

I’ve calculated the free memory, for the part of generating password normally needs 330 to 360 bytes memory. Would there be a problem if it got 419 bytes memory left at the stage of generating password?

Libraries I’ve included:
#include <SPI.h>
#include <Ethernet2.h>
#include <PubSubClient.h>
#include <sha1.h>
#include <Base64.h>
#include “Authentication_OneFile.h”
Which PubSubClient is from knolleary’s, SHA is from Cryptosuite, and Base64 is from adamvr

Environment:

  • Arduino UNO R3
  • Ethernet shield 2
  • IDE 1.7.4

Libraries I’ve used are also attached. I did my best to optimize Auth/Authentication_OneFile.cpp
No sure if there is any place can be improved…

Suggestions would be appreciated.

mqtt_test.ino (2.61 KB)

libraries.zip (347 KB)

I only looked at the .ino file. That looks like a sensibly-constructed test program, with F() macro used everywhere that it's appropriate. If you are having difficulty fitting that into memory then there's either something wrong with the libraries consuming too much space or you just simply have to have a bigger Arduino to do this Ethernet stuff.

Hey.

Firstly, just try it.

If it works then yeah.

If you want memory optimisations...someone taught me a few yesterday which were a big help:

  1. Move any strings to program memory using the F macro:
Serial.print(F("This string is now in the program memory and not the SRAM")
  1. If you have any variables that you get from a user that are numbers 256 or less...use the byte type and not the int type. This saves a byte or so per vairable...or if using an array it can save quite a lot!
int i = 12; //Uses 2 bytes of SRAM
byte i = 0b001100; // Uses only a single byte of SRAM

Better still...if you ask basic questions for setting up like "Do you want to use 9600 baud?" and also maybe 7 other "closed yes/no" questions... you could stick all of them in a byte and do operations on the byte type later.

This will take the 8 ints (16 bytes) down to like 1.

Actually...just looked at your .ino and it seems you have done as many optimisations as you can yourself already...

Maybe only checking the libraries and seeing if they can be further optimised may help?

Network connectivity, and doing "simple" network related tasks as you're doing, really eats through space. These protocols were not made with devices with a mere 2k of ram and 32k of program memory in mind. Looking at the laundry list of libraries you're using, I'm impressed you've made it fit at all.

You may need to use a larger chip - I'd pick the 1284p if I was making an internet connected arduino project (for internet connected projects, I usually use an Espruino, which runs on an ARM processor and is programmed in JS); the 1284p has the most RAM of the atmega series, plus 128k of flash. It's a really solid chip.

332 bytes of password? Why on earth?

Google calculator reports the value as infinity.

A small correction to your rhetorical question... Why in the observable, unobservable, and imagined universe?

Changed to a mega, it worked then.

needs 330 to 360 bytes memory. Would there be a problem if it got 419 bytes memory left at the stage of generating password?

Don't forget that you'll need some (not very well defined, for libraries) memory for stack usage, plus malloc overhead (assuming that you're using malloc.)

Did you get memory usage numbers from the working code on a MEGA? It would be interesting to "pre-fill" the free memory with a known constant, so you can scan after some time to see how much of it has been used at any point, rather than how much is in use at a particular point. The lack of visibility into stack and malloc usage in avr-gcc is ... annoying.