Instability with large programs

Using Arduino Programming Questions
#1

Hi experts,

I have a fairly large program running on an Atmega328P, close to 4000 lines of code, 31676 bytes (96% of capacity). Global variables use 37% of memory.

The program interfaces an SD card reader, an RTC, and a bluetooth module.

As I approached 96% of code size, I have noticed that behaviour in general has become increasingly erratic.

Specifically, I have noticed that small snippets of code, that aren't called at runtime (would only be activated by a sequence of events e.g. button presses, bluetooth commands etc.) either make or break whether or not the program executes at all. By this I mean it won't even execute the first command in setup.

Some of this behaviour seems to be related to the bluetooth (via SoftwareSerial) - behaviour stabilised somewhat when I reduced or eliminated entirely the debugging output to bluetooth.

However there is some erratic behaviour that can't be explained by that (all bluetooth output switched off).

I am using FreeRAM to monitor memory usage. As far as I can see, it isn't a problem. When I comment out bits of code that seem to make or break the execution, the FreeRAM result is the same regardless.

Posting the code isn't really viable or useful - it's almost 4000 lines and these strange problems only started happening once I approached this 95%+ size.

If I cut back the code by commenting out functions that aren't called unless a certain sequence of events occurs (i.e. only by button presses, bluetooth commands etc. they are not called at setup / loop), the program stabilises and will start (by this I mean it will run the first command in setup). I have verified that they are NOT being called. This behaviour I can't explain at all.

My question is: does a large program size cause stability issues?

Under what circumstances would code that isn't called affect whether or not the first command in setup is executed?

I can't see why it would be a problem, however I can't explain the strange behaviour of my code.

Not much to go on sorry, and I fully expect to be flamed for this because it is vague, but any suggestions very much welcome.

#2

No, a large program size does not cause instability.
Use of RAM you don't have causes instability.

#3

Do you want the best help? Post your code. We might spot the one place you are corrupting memory, the most common problem.

Are you using String?

Are you (or your libraries) using malloc/free?

And instead of SoftwareSerial, you should consider using AltSoftSerial on pins 8 & 9. If you can't switch pins, take a look at my NeoSWSerial. Both of these alternatives are much more efficient than SoftwareSerial. SoftwareSerial disables interrupts for long periods of time, forcing your 328 to twiddle its thumbs while each character arrives. At 9600 baud, it could have executed 10000 instructions instead of waiting for each character. :-/

Both libraries are available from the Arduino IDE Library Manager, under the menu Sketch -> Include Library -> Manage Libraries.

#4

AWOL nailed it. Here's a six page thread that should give you some insight on why FreeRAM results mean little to nothing with regard to stability issues.

#5

I have sketches that use 90% of available RAM. They have never failed for that reason.

#6

rj77:
My question is: does a large program size cause stability issues?

Absolutely NOT! You can use every last byte of program space, and it will have ZERO effect on stability.

rj77:
Under what circumstances would code that isn't called affect whether or not the first command in setup is executed?

Under the circumstance where you have bugs in your code...
Regards,
Ray L.

#7

-dev:
Do you want the best help? Post your code. We might spot the one place you are corrupting memory, the most common problem.

Are you using String?

Are you (or your libraries) using malloc/free?

And instead of SoftwareSerial, you should consider using AltSoftSerial on pins 8 & 9. If you can’t switch pins, take a look at my NeoSWSerial. Both of these alternatives are much more efficient than SoftwareSerial. SoftwareSerial disables interrupts for long periods of time, forcing your 328 to twiddle its thumbs while each character arrives. At 9600 baud, it could have executed 10000 instructions instead of waiting for each character. :-/

Both libraries are available from the Arduino IDE Library Manager, under the menu Sketch → Include Library → Manage Libraries.

Thanks very much for the tips. I’m not using malloc/free or String, not sure if any of my libraries are - they are all pretty standard, but I will do a text search.

I will definitely try an alternative to SoftwareSerial since disabling this often brings everything back to life (until I try to debug again!)

One thing I am doing is calling SD (SDFat.h library) from within functions rather than as a global to free up RAM. It uses a huge amount of RAM and FreeRAM (I understand it’s not very useful) seems to recognise this when I call it from within the SD functions… however I will read the 6 page thread to look for insights. Perhaps it’s too much strain on the heap? (for want of a better phrase)

If I don’t have any joy I will try to post a summarised version of the code since I think it’s way too much to ask for someone to wade through 4000 lines.

Thanks again

AWOL:
No, a large program size does not cause instability.
Use of RAM you don’t have causes instability.

That’s what I always thought but I figured it was worth asking anyway…

So regardless of if the snippet is called or not, if it is incorrect, the entire program will fail to execute even one line? (assuming -fpermissive on the compiler has or hasn’t ignored it?)

#8

So regardless of if the snippet is called or not, if it is incorrect, the entire program will fail to execute even one line?

Code that can never be called is NOT uploaded to the Arduino.

Code that is not syntactically valid is not compiled, so the upload will fail.

Code that could be, under some conditions, called, that is logically incorrect, will not affect other code until it IS called.

It is far better, when posting here, to NOT wave your arms. Use that energy to post your code.

#9

PaulS:
It is far better, when posting here, to NOT wave your arms. Use that energy to post your code.

I can't help but wonder why there is as much arm-waving on this forum as there is.

#10

The next thing I tried was to remove everything from loop() but keep setup() the same.

When I take everything out of loop() and just leave setup(), everything in setup() works every single time I power it on.

With everything still in loop(), the program sometimes executes setup() on power on, and sometimes doesn't. It seems completely random, doesn't matter how quickly I power it off/on, or if I leave it for ages. If it starts setup(), then everything (including loop) works absolutely fine. So it either all works or nothing works i.e. if the first command works in setup, then the rest will work.

I guess what I am trying to grapple with is why the contents of loop() and the functions that it calls makes any difference to whether setup() begins to execute or not.

#11

Time to post your code. Either in full or a representative example that exhibits the behaviour.

#12

OK, I should have pointed out that I am uploading using a USBasp and I am not using the bootloader.

after advice as per: Reclaiming some Program Memory from the bootloader - Microcontrollers - Arduino Forum

boards.txt has been changed so that the maximum sketch size has been increased from 30720 to 32768 (using Barebones ATmega Chips (no bootloader) method)

After I changed my code to the following, I have found that it starts to behave badly when the sketch length exceeds 30720.

Huge apologies for not putting this in the original post, that was a massive oversight.

Is there any way that I can use >30720 with stability?

Thanks very much

void setup() {
  pinMode(9, OUTPUT);
  digitalWrite(9,HIGH); 
  delay(3000); 
  digitalWrite(9,LOW); 
  delay(3000); 
  digitalWrite(9,HIGH); 
  delay(3000); 
  digitalWrite(9,LOW); 

  // now copy paste delay(3000); until the sketch length approaches 100%
  delay(3000); // hundreds of times
}
  
void loop() {

}
#13

OK, took the challenge. Using a standard Sparkfun Redboard and IDE 1.6.6

Sketch uses 30,712 bytes (95%) of program storage space. Maximum is 32,256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2,039 bytes for local variables. Maximum is 2,048 bytes.

That works as expected.

Added one extra delay (total 2476 delays including those for the blink)

Sketch uses 30,724 bytes (95%) of program storage space. Maximum is 32,256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2,039 bytes for local variables. Maximum is 2,048 bytes.

And the verification by avrdude fails at memory location 0x7800 (that is 30720 decimal) and the board is 'dead'.

Reading | ################################################## | 100% 10.35s

avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x7800
         0xff != 0xf8
avrdude: verification error; content mismatch

avrdude done.  Thank you.

I would suggest that you make sure that you're not exceeding whichever limit there is.

#14

sterretje:
And the verification by avrdude fails at memory location 0x7800 (that is 30720 decimal) and the board is 'dead'.

Reading | ################################################## | 100% 10.35s

avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x7800
        0xff != 0xf8
avrdude: verification error; content mismatch

avrdude done.  Thank you.



I would suggest that you make sure that you're not exceeding whichever limit there is.

Hi, thanks so much for "taking the challenge"!

However, when I do a read verify, I get:

avrdude: 31032 bytes of flash written

avrdude: verifying ...
avrdude: 31032 bytes of flash verified

avrdude done.  Thank you.

So something is different - as far as avrdude is concerned, everything is fine.
Is there some flag I need to set? I was under the impression that I can impinge on the bootloader's space if I don't need it since I am using the USBasp to program it (and bluetooth to communicate)

#15

No idea; this is now outside my knowledge.

#16

sterretje:
No idea; this is now outside my knowledge.

Sincere thanks for doing the experiment, I really appreciate it.

If anyone else can weigh in with any insights, that would be great.

Thanks very much

#17

I should have pointed out that I am uploading using a USBasp and I am not using the bootloader.

after advice as per: Reclaiming some Program Memory from the bootloader - Microcontrollers - Arduino Forum

boards.txt has been changed so that the maximum sketch size has been increased from 30720 to 32768 (using Barebones ATmega Chips (no bootloader) method)

Did you reprogram the fuses to turn off the "BOOTRST" fuse? Otherwise, excecution will start at the bootloader address whether the bootloader is present or not, causing ... nothing good, and probably really bad.

#18

Thank you, I was hoping that the Barebones boards addon would do that but I guess not - would you mind telling me how to do this?

#19

OK, worked it out, used the AVR fuse calculator and set the BOOTRST.

Thanks to everyone for help!

#20

My question is: does a large program size cause stability issues?

No.

But if you do have bugs there may be more chance of you noticing them in a larger program.

Let's say you have a minor memory leak in one of your libraries - for instance writing 10 characters to an array that's only sized for 8. If you are using that library in a small program then it maybe that although that code is overflowing and corrupting something it might not cause problems because that memory isn't being used by anything else anyway. So you test that library and you think it works fine.

Then you compile that same library into project that uses 90% of available memory - there's now a lot more chance that the two-byte overflow that didn't cause any harm before is now corrupting something important.