Reducing Sketch Size - specific pointers needed

Hi

I have a sketch that has grown to a reasonable size, and needs to have more functionality included - but I am up to 95% of program storage space used (27424 out of 28672) on a Leo Eth. So I am looking to improve my code before continuing to expand it.

I know the forum has many ‘sketch size’ posts and I have read a few of them, tried to follow the advice, but only managed tiny reductions (about 300 Bytes reduction in total!).

What I have tried and worked:

  • Control the port directly - done, I now use DDRB, PORTB, or’s and and’s and all works - saved about 20 bytes!
  • Reduce variables to lowest needed sizes - done, I now use boolean / byte / word rather than int - saved a few more bytes.

What I have tried and didn’t work:

  • Reducing the size of library’s. I renamed Base64 to myBase64, removed the decode functions - didn’t save any space as I assume the compiler optimises them out anyway. Tried changing variables from int to byte - got my self stuck in ‘byte’ and then ‘uint8_t’ “does not name a type” errors that I could not resolve.

What I think I should try but don’t know how:

  • I shouldn’t use String’s, I should use char x - but I am very confused about the allocation of memory. If I don’t know how long my string is going to be that I need to store, how do I know what size to declare it as? I have used a couple of char x's but have declared fairly big buffers and have not put any code in to check for overruns which may cause me issues?

  • I have some long static strings in F(""), I would like to keep them as it makes for good debugability (!), but any way to help the compiler store them more efficiently? For example I use the function names very often in the strings, but defining a const char:
    const char setupStr = “setup() “;
    and then using that multiple times:
    SERIALOUT << F(”\r\n”) << setupStr << F("-- Send Email based on GPIO changes --") << endl;
    take up more space than a straight print:
    SERIALOUT.println(F("\r\nsetup() – Send Email based on GPIO changes --"));

  • I shouldn’t overload the print and println functions with different variable types, but I have tried using sprintf to move things to a string first uses even more code?

I know I could move to a mega - but I like the Leo, having Ethernet and SD card on the main board makes for a smart installation.

Full code and compiler output attached.

Thanks very much for your help in advance.

Kevin

SendEmail_08.ino (18.1 KB)

Compiler_output.txt (24.1 KB)

Do you REALLY need such long debug messages? Does "setup() Failed to start Ethernet using DHCP." really convey more than "DHCP Fail"? Does the former require more code space than the latter?

Is using the Streaming library REALLY value added? Yes, it makes getting data out easier, but it is not necessary. The same output can be generated using multiple print() statements.

A static string (char array) will take up no more SRAM than a String instance containing the same data. The only issue is choosing a suitable size for the array. You control the names and values that the settings can have. You, therefore, know the lengths of the strings that can be used for the names, and can, therefore, statically size an array appropriately.

The String class has toFloat() and toInt() methods. The toInt() method is stupidly named, since it returns a long. The methods that you have, that copy the data in the String instance, are not needed.

For example I use the function names very often in the strings, but defining a const char:
    const char setupStr[] = "setup() ";

You're wasting RAM there.

AWOL - Thanks for the reply, but it isn't very helpful to me....

Would you be able to enlighten me as to how I can store the strings whilst not wasting RAM, or more relevantly at the moment, in the most Program Memory efficient way?

Thanks very much
Kev

The string I quoted is already in program memory, but is copied into RAM during crt0.
A PROGMEM qualifier will prevent that.

Your code itself is pretty small...

I'd say your problem is that you want to use a bunch of rather large libraries (internet connectivity, SD card, and email - all of which are fairly large), and cram it all into a chip with 32k of flash. A mega or something based on a '1284p would probably be more appropriate for this project. (there aren't any official 1284p boards, but it's pretty well supported with MightyCore and various third-party boards, or you can build standalone 1284p on perfboard).

Note that you only need to reduce the program size if you're up against the limit - you can use every last byte of program memory without issue (unlike RAM, where you need to leave space for the stack and local variables)

Hi Paul

Thanks very much for your thoughts.

PaulS:
Do you REALLY need such long debug messages? Does "setup() Failed to start Ethernet using DHCP." really convey more than "DHCP Fail"? Does the former require more code space than the latter?

Your right, they are a bit long, shortened and that saved me 360 Bytes.

PaulS:
Is using the Streaming library REALLY value added? Yes, it makes getting data out easier, but it is not necessary. The same output can be generated using multiple print() statements.

I agree it is a 'nice to have', but it uses no Program or SRAM space - I have just tested it on a standalone piece of code to confirm.

PaulS:
A static string (char array) will take up no more SRAM than a String instance containing the same data. The only issue is choosing a suitable size for the array. You control the names and values that the settings can have. You, therefore, know the lengths of the strings that can be used for the names, and can, therefore, statically size an array appropriately.

Setting Names I agree, I control, but I can't control the values that the settings have - that is down to what a user writes in the settings.txt file, unless I am miss understanding you?

PaulS:
The String class has toFloat() and toInt() methods. The toInt() method is stupidly named, since it returns a long. The methods that you have, that copy the data in the String instance, are not needed.

Thanks, I have commented out toFloat and toLong as I don't use them (but the compiler optimised them out anyway). The toBoolean I do use so have left in.

Any further thoughts welcomed as I am not making much progress - 1% so far.

Thanks very much
Kev

AWOL:
The string I quoted is already in program memory, but is copied into RAM during crt0.
A PROGMEM qualifier will prevent that.

Thanks AWOL, so I assumed that:

PROGMEM const char oBP = "onButtonPressed() ";

Serial1 << oBP << F(“Send Email 8”) << endl;
Serial1 << oBP << F(“Email 8 failed”) << endl

would save me program memory over:

Serial1.println(F(“onButtonPressed() Send Email 8”));
Serial1.println(F(“onButtonPressed() Email 8 failed”));

as onButtonPressed() is only stored in program memory once - however the println version is over 100 Bytes smaller !?!?

Thanks
Kev

KevWal:
however the println version is over 100 Bytes smaller !?!?

What's the rest of the code look like? Does the rest of the code use the Streaming library? That library needs some space for its code. If nothing else there uses it then when you use the println version it gets optimized away.

I would kill that Streaming library with fire.

Not only does it waste space when you use it (if you include but don’t use it, the compiler will optimize out the code), the coding style it promotes is an abomination; it hurts my eyes.

You will almost always win with print/println calls, and the overhead for multiple print() calls is very small.

KevWal:
Any further thoughts welcomed as I am not making much progress - 1% so far.

change/upgrade platform! A Particle Photon is Arduino-like, has a ton of working memory, a real DAC, built in Wi-Fi, access to a richer subset of C++, and for tinkerers has a great set of free tools like Publish/Subscribe methods and web-hooks, all integral to its basic implementation.

Relax, stretch out your legs and go nuts!

USD19.00 (of course you have to add your own SD card for $3.00 or so). Think of it like an Arduino that's advanced in the past three years. If you really need ethernet, well they support Pi as well.

Or, there are other Arduino-like platforms that offer more breathing room as well... Check out Adafruit.

I wouldn't step outside the AVR world without a good reason - library support on non-AVR platforms is pretty crap, and he's just short on flash. The AVR solutions I suggested above would be less expensive, have better library support than the photon, and would provide enough flash for him..

DrAzzy:
I wouldn't step outside the AVR world without a good reason - library support on non-AVR platforms is pretty crap, and he's just short on flash. The AVR solutions I suggested above would be less expensive, have better library support than the photon, and would provide enough flash for him..

Humbug!

Just poke your head out of the tent... LOTS of support including most of the most common/popular Arduino libraries.

Particle's Arduino heritage extends into its supported libraries (most port quite easily).

And then there is also the form factor, which is very efficient/flexible.

PS. I wasn't at all trying to impugn your suggestions, by the way.

BulldogLowell:
change/upgrade platform! A Particle Photon is Arduino-like, has a ton of working memory, a real DAC, built in Wi-Fi, access to a richer subset of C++, and for tinkerers has a great set of free tools like Publish/Subscribe methods and web-hooks, all integral to its basic implementation.

Relax, stretch out your legs and go nuts!

USD19.00 (of course you have to add your own SD card for $3.00 or so). Think of it like an Arduino that's advanced in the past three years. If you really need ethernet, well they support Pi as well.

Or, there are other Arduino-like platforms that offer more breathing room as well... Check out Adafruit.

Thanks Bulldog - I need physical Ethernet and have put a lot of time in to the code so far - so not really looking to move away from Arduinio and the supportive community surrounding it.

Cheers
Kev

Hi All

Thanks for your suggestions, I really do appreciate them.

Hardware wise I can move if absolutely needed, but there are benefits to sticking to what I have if I can make it work.

Can I ask some specific questions about things I have seen, can people help me out on things like:

  • Not overloading the println function?

  • Allowing "F("onButtonPressed() "to be stored in PROGMEM once, rather than the 10 times it is currently?

  • Not using Strings

  • Removing Keyboard / Mouse support from the board?

  • Removing USB Serial Support from the board?

  • Using a smaller SD card library?

Thanks very much
Kevin

KevWal:
Hi All

Thanks for your suggestions, I really do appreciate them.

Hardware wise I can move if absolutely needed, but there are benefits to sticking to what I have if I can make it work.

Can I ask some specific questions about things I have seen, can people help me out on things like:

  • Not overloading the println function?
  • Allowing "F("onButtonPressed() "to be stored in PROGMEM once, rather than the 10 times it is currently?
  • Not using Strings
  • Removing Keyboard / Mouse support from the board?
  • Removing USB Serial Support from the board?
  • Using a smaller SD card library?

Thanks very much
Kevin

I took a quick look and it appears that a lot of the activity is Serial Debug, and pretty wordy at that:

    case 0:   break; // SERIALOUT.println(F("Ethernet.maintain() Nothing needed"));
    case 1:   SERIALOUT.println(F("eMaintain() Renew failed")); softReset(); break;
    case 2:   SERIALOUT.println(F("eMaintain() Renew success")); break;
    case 3:   SERIALOUT.println(F("eMaintain() Rebind fail")); softReset(); break;
    case 4:   SERIALOUT.println(F("eMaintain() Rebind success")); break;
    default:  SERIALOUT.println(F("eMaintain() Unexpected number")); softReset(); break;

a simple approach would be to eliminate all of that... or you can use a macro like this to optionally print the debug:

#define DEBUG // comment me out to disable Serial Debug
#ifdef DEBUG
#define DEBUG_BEGIN(x) Serial.begin(x) // in your case you are using Serial1
#define DEBUG_PRINT(x) Serial.print(x)
#define DEBUG_PRINTLN(x) Serial.println(x)
#else
#define DEBUG_BEGIN(x)
#define DEBUG_PRINT(x)
#define DEBUG_PRINTLN(x)
#endif


void setup() 
{
  DEBUG_BEGIN(9600);
  DEBUG_PRINTLN(F("Hello World"));
}

void loop() 
{

}

Also, you are right... if you are sending the same message in your code in 10 different places, well store that message in PROGMEM and retrieve the messages accordingly .

But start with jettisoning your Serial Debug and see what you are saving. Let us know!

Hi

So, since my last post I have

A) Added more of the program functionality - which has taken me up to 99% space used

B) Implemented the #ifdef DEBUG functionality on all the serial prints.

With DEBUG not defined the code reduces to 85% PROGMEM used. However, this code is going to go out in the wild in systems that I wont have access to - so I really need the debug functionality to be left in so that others can try and fix issues (like settings file issues, not dhcp'ing an IP, etc) or so that they can report back to me accurately where the problem is and I can debug the issue remotely.

In fact the remaining features I want to add are around more debug functionality - 1) copying all of the log messages to a txt file on the SD card for example so that they can send me the file and I can see, and 2) adding a screen and outputting the messages there too.

Help and ideas on space efficient ways of taking the same message from PROGMEM into both the Serial outout, and LCD and a file appreciated?

And also, back to the options I have to reduce the size of code. Any help on the following, or other ideas would be much appreciated:

  • Not overloading the println function?
  • Allowing "F("onButtonPressed() " etc to be stored in PROGMEM once, rather than the 10 times it is currently?
  • Not using Strings
  • Removing Keyboard / Mouse support from the board?
  • Removing USB Serial Support from the board?
  • Removing the boot loader?
  • Using a smaller SD card library?

Thanks very much
Kevin

KevWal:
Hi

So, since my last post I have

A) Added more of the program functionality - which has taken me up to 99% space used

B) Implemented the #ifdef DEBUG functionality on all the serial prints.

… However, this code is going to go out in the wild in systems that I wont have access to - so I really need the debug functionality to be left in so that others can try and fix issues (like settings file issues, not dhcp’ing an IP, etc) or so that they can report back to me accurately where the problem is and I can debug the issue remotely…

  • Removing USB Serial Support from the board?

I guess I though you could ditch all that Serial Debug if you were seriously considering tossing USB Serial Support.

if you right brain and left brain can agree and you need Serial, you can try to standardize/condense all of your Serial output to readable, minimum output size.

You don’t need to be so verbose if you can really afford not to.

D:14,
F:17,
A:13,

etc…

Saving to SD cards isn’t debug, that’s data storage…