EDIT:
Long story short, I was having SRAM overflows because my code was eating up all the memory (used String instead of char[], and also a lot of Serial.print("some text").
If you really need it, you CAN have a 512 byte long NewSoftSerial buffer without issues, as long as you optimize memory usage.The solution:
Arduino v0.23:
- Create one or more char as global buffers - one of them being used for Serial.print().
- Store your Serial.print() messages in PROGMEM, see post below by draythomp).
UPDATE 2012-06-14 >> Arduino v1.0+: Use Serial.print(F("some text")) instead - this stores your strings in PROGMEM and automatically calls them without the need of an SRAM buffer).[/quote]
Hi there,
I'm working with a TTL device (through the NewSoftSerial Library) that sometimes sends large bursts of data (@9600bps), and I'm not able to read them quickly enough to avoid buffer overflow.
I thought the best workaround would be increasing the buffer on the software serial port.
The buffer size is defined in NewSoftSerial.h:
#define _NewSS_MAX_RX_BUFF 64 // RX buffer size
Default is 64 bytes, this value can be increased by the user before compiling, effectively resulting in a larger buffer.Problem is, while coding, compiling and testing, at times my program would suddenly stop working - although compiling successfully (Arduino would either output unexpected chars and lock right at the start, or exhibit that behavior after some seconds).
Because I'm new to the whole micro-controller thing and I like to get my hands dirty before reading properly, it took me a week to even consider that the problem was most probably due to an SRAM overflow.
That said, I started to mess around with the allocations for char arrays and NewSoftSerial buffer, and voila. There it is working again.So, care to share any ideas on how to work around this?
I need a buffer of at least 350 bytes, and a char array of at least 200 bytes to free the buffer and process its messages.Thanks in advance for reading. Will appreciate any input you may have on the subject.
Sorry, forgot to mention:
Using Arduino Uno, with ~23 global variables at the moment:
3 x int[3]
2 x int
2 x unsigned int
2 x unsigned long
10 x boolean
1 x char
3 x String
I'm currently using a fixed global char[200] to read from NewSoftSerial, because it "feels like" using String doesn't "seem to" help with memory stability. (lots of double quotes here)
I also wanted to use MsTimer2.h to trigger 3 LED's states (between off, on, blinking slow and blinking fast), but then again I think I shouldn't use that because it will mess up NewSoftSerial's interrupt-driven operation.
And finally I'm wondering if using Metro.h would trigger some sort of incompatibility, because I stopped using it when I disabled MsTimer2. Now when I try to use it without MsTimer2, the program doesn't run correctly, but I'm still unsure if it's a memory-related problem or not...
I'm currently using a fixed global char[200] to read from NewSoftSerial, because it "feels like" using String doesn't "seem to" help with memory stability.
The best thing that you can do is get rid of all String objects. Look at the String class. Specifically, look at the constructor and the += operator functions. A String is allocated EXACTLY large enough to hold the initial string. When a character is added to the String, more memory is allocated, to hold the initial string plus the new string (usually one character). The data in the old space is copied to the new space. The data to be appended is copied to the new space. Then the old space is freed.
When the next character to be added, the space used by the String object before is a little too small for the new String, resulting in small blocks of space all over the place not being large enough to reuse, but not being contiguous, either.
C/C++ has no mechanism to relocate allocated memory, so eventually, (pretty quickly, sometimes), you no longer have enough memory for malloc to succeed. Things get ugly when this happens.
On a UNO, 512 bytes for a buffer is 1/4 of SRAM. If you have much else going on, reserving that much for NewSoftSerial to use as a buffer is not a great idea.
Better would be to make sure that you are pulling data out of the buffer faster than the other device is making NewSoftSerial fill it.
Of course, without seeing your code, best wishes is all we can offer. So, good luck.
@PaulS,
Thank you for explaining.
You're right, it's lazy of me not to provide some code to look at.
I'm writing a workable example - it's taking me more time than I anticipated.
The thing with String is that it comes with a lot of useful built-in functions. Maybe char arrays have them as well, it's just that String functions are easy to use and fairly documented and explained in this site.
How about moving these global char arrays and the NewSoftSerial buffer to PROGMEM? With Mikal Hart's Flash Library. Or is it read-only once written at boot-time?
Better would be to make sure that you are pulling data out of the buffer faster than the other device is making NewSoftSerial fill it.
That's precisely my issue. The shield outputs too much data "at once" for the Arduino to process it in time, e.g. it outputs two or more consecutive response lines, and that doesn't give the Arduino enough time to read and process one line before reading the next.
I tried to use the shield @4800bps, but at that speed I'm getting some ÿ chars, and I'm using if(shield.available()) before shield.read().
But that's all talk and no game, so please bear with me until I post example code.
Thanks again.
To keep track of your memory usage:
#include <MemoryFree.h>
Then sprinkle some Serial.println(freeMemory()); around in places to see what's going on. You need to keep a couple of hundred bytes unused to allow for subroutine calls, temporary variables and such. This will verify your problem and allow you to focus. Then take a look at your usage of "some text" constants. These eat up memory more quickly than you can imagine. To get around this, look at PROGMEM (google it).
PaulS's explanation about memory fragmentation is right on and easily understood. Yes, the Streams library is cool and yes it's easy to use and yes, it costs a whole lot in memory use in payment for that flexibility. Take a look a the standard C library string manipulation routines, you can save a whole lot of ram using the more primitive and simple techniques. Even the oft maligned sprintf with a hundred byte buffer will save you tons of ram over the usage of Strings.
good luck
How about moving these global char arrays
What char arrays?
and the NewSoftSerial buffer to PROGMEM?
No. PROGMEM is read only. A serial buffer that was read only would be pretty much useless, wouldn't it?
At 9600 baud there is quite some time before you collect 500 chars (for a processor). Can you post your sketch to see if we can improve on the efficiency?
draythomp, you're a lifesaver.
Because you made it so easy to diagnose the origin of the problem. Thank you.
More on that subject, I'm a total ignorant.
The memory "leak" comes from my Serial.print() calls for debugging and sending commands to the shield.
I really had no idea that placing strings in Serial.print("some string") would allocate them ALL in SRAM at boot-time.
I thought they were temporary, i.e. only placed in SRAM while printing, and then cleared.
It made so much sense to me because they already occupy space in program memory.
Only now am I starting to understand the whole point of storing read-only strings in PROGMEM.
PROGMEM scares the hell out of me. I was afraid this day would come, eventually.
@PaulS, I'm sorry, you didn't understand my question because I didn't understand that PROGMEM was read-only (thus my question wasn't logical).
Now, to the other underlying issue (NewSoftSerial buffer overflow), here's the code:
#include <MemoryFree.h>
//#include <MsTimer2.h> //--- // uses interrupts, incompatible with NewSoftSerial
#include <Metro.h> //--- // is this incompatible with NewSoftSerial?
#include <NewSoftSerial.h> // NewSoftSerial v10c
// had to define the buffer size to 450 to prevent buffer overflow
// (changed the NewSoftSerial.h file: #define _NewSS_MAX_RX_BUFF 450 // RX buffer size)
#define ATBUFFSIZE 200 // char mode - this sets the global char array size
unsigned long timer_memfree = 0;
int LED_RYG[3]={10,11,12}; // led pins
int LED_RYG_STATE[3]={HIGH,LOW,LOW}; // stores current state of leds
int LED_RYG_SET[3] = {1,0,0}; // stores desired state of leds (also initial state)
// call LED_SetAll(LED1_STATUS,LED2_STATUS,LED3_STATUS,FORCEUPDATE) to set LED status
// 0 - LED off
// 1 - LED on
// 2 - slow blink (500ms)
// 3 - fast blink (150ms)
Metro LED_BLINK_SLOW = Metro(500,1); //--- // 500ms with autoreset (ignore missed events)
Metro LED_BLINK_FAST = Metro(150,1); //---
char ATBuff[ATBUFFSIZE];
unsigned int nBufPos;
//
// plus 20 other global variables (4 bytes max. each)
//
NewSoftSerial shield(2,3); // NewSoftSerial v10c
//SoftwareSerial shield(2,3); // NewSoftSerial v11 beta
void setup()
{
// Arduino's input/output pins:
for (int i=0; i<3; i++)
{
digitalWrite(LED_RYG[i],LED_RYG_STATE[i]); // sets initial state of output LEDs
pinMode(LED_RYG[i],OUTPUT); // sets operation mode for LED pins (output)
}
//MsTimer2::set(50, LED_REFRESH); //--- // incompatible with NewSoftSerial
//MsTimer2::start(); //--- // incompatible with NewSoftSerial
Serial.begin(9600); // hardware serial port (arduino)
Serial.println("-");
Serial.print("NewSoftSerial v");
Serial.print(NewSoftSerial::library_version()); // NewSoftSerial v10c
//Serial.print(SoftwareSerial::library_version()); // NewSoftSerial v11 beta
Serial.print(", buffer ");
Serial.print(_NewSS_MAX_RX_BUFF); // NewSoftSerial v10c
//Serial.print(_SS_MAX_RX_BUFF); // NewSoftSerial v11 beta
Serial.println(" bytes");
Serial.println("> Starting Shield...");
shield.begin(9600); // software emulated serial port (arduino pins 2,3)
}
void loop() {
MemCheck(0); // checks free memory every 1000 ms
// if a character is sent from the terminal to the Arduino...
if(Serial.available() > 0)
{
shield.print(char(Serial.read())); // send the character to the shield
}
// if a character is coming from the shield...
while (shield.available() > 0)
{
ATGetResp(); // read the whole line
MemCheck(1);
LED_REFRESH();
ATParse(); // parse the line and do the necessary operations before proceeding
MemCheck(1);
LED_REFRESH();
}
}
void MemCheck(boolean force_check)
{
if (millis() - timer_memfree > 2000 || force_check)
{
Serial.print("Free memory: ");
Serial.print(freeMemory());
Serial.println(" bytes");
timer_memfree = millis();
}
}
// This function gets a response from the shield
// and stores it in the global char array, ATBuff.
// ATBuff is replaced with the new response every time this function is called
// (or null char array if there's no response).
// The function returns when a complete line is received (lines end with CR+LF).
void ATGetResp() {
char c='\0'; // temporary char for storing shield.read()
memset(ATBuff, '\0', ATBUFFSIZE); // clear response array (char mode)
nBufPos = 0; // Reset array counter (char mode)
//ATBuff = ""; // clear the response line (string mode)
unsigned long timer_ATGetResp = millis();
while (millis() - timer_ATGetResp < 3000) // will give up collecting data after 3 seconds
{
// check for NewSoftSerial buffer overflow
if(shield.overflow() == true)
{
Serial.println(">> ERROR: soft serial buffer overflow!");
}
// process incoming serial data in the software serial port
if (shield.available() > 0)
{
c = shield.read();
ATBuff[nBufPos++] = c; // char mode
//ATBuff += c; // string mode
if (c == '\r')
{ // if c is a carriage return, the response line ended. Don't save CR to the string.
// Read and ignore the next character - should always be '\n' (newline character).
c = shield.read();
if (c == '\n')
{
ATBuff[nBufPos] = '\0';
return;
}
// if it's not '\n', return anyways,
// BUT THERE WILL BE A LOST CHARACTER!!!
// Technically this should never happen.
// For debugging purposes:
Serial.print("> ERROR: Losing a char here!!! Char='");
Serial.print(c);
Serial.println("'");
ATBuff[nBufPos] = '\0';
return;
}
}
// check for global char array overflow
if (nBufPos > ATBUFFSIZE - 3)
{
Serial.println("> ERROR: ATBuff[] overflow!");
ATBuff[nBufPos] = '\0';
return;
}
}
ATBuff[nBufPos] = '\0';
return;
}
///////////////////////////////////////////////////////
void ATParse(){
//if (ATBuff.length() < 2) // string mode
if (nBufPos < 3) // char mode
{ // ignore empty lines
//Serial.print("> Response ignored: "); // debug
//Serial.println(ATBuff); // debug
return;
}
Serial.print("Response: ");
Serial.println(ATBuff);
if (strstr(ATBuff, "+RESP01") != 0) // char mode
{
Serial.println("Response: +RESP01");
// update global variables
// parse other portions of ATBuff[], etc.
return;
}
//
// and some 16 additional if statements with similar operations
// --- THIS IS WHERE I'M EATING UP ALL THE MEMORY ---
// All the strstr(ATBuff, "response")
// and Serial.print("some string") are being stored in SRAM!
// they're allocating around 1KB of SRAM at boot-time
//
//if (ATBuff.length() < 4 && ATBuff.indexOf("OK") != -1) // string mode
if (nBufPos < 5 && strstr(ATBuff, "OK") != 0) // char mode
{
Serial.println("Response: OK");
return;
}
//if (ATBuff.length() > 2) // string mode
if (nBufPos > 3) // char mode
{
Serial.print("> Unknown message: ");
Serial.println(ATBuff);
return;
}
}
///////////////////////////////////////////////////////
// This function was called periodically by MsTimer2.
// Due to incompatibility between MsTimer2 and NewSoftSerial,
// I will need to call the function several times along the code.
// Ideally, the period should be, at most, half the smallest interval
// between the Metro instances (so that the blinking intervals
// appear to be consistent/natural).
void LED_REFRESH()
{
// some code
}
//////////////////////////////////////////////////////
// Allows the user to set all the LED status at once,
// and choose if they refresh to the new status instantly
// or just after the next MsTimer2 occurrence
void LED_RYG_SetAll(int STATE_RED, int STATE_YELLOW, int STATE_GREEN, boolean FORCE_REFRESH)
{
// some code
}
I don't see a way of reading it faster...
Once again, thank you.
footswitch
It made so much sense to me because they already occupy space in program memory.
Actually, they don't. Look at a disassembly view if you are really interested, but what is in program memory is just a pointer to the data in SRAM.
You could create an array for each value that you want to use in strstr, and store those arrays in PROGMEM.
@PaulS
what is in program memory is just a pointer to the data in SRAM.
What I mean is that uploading the sketch will store the strings in program memory in some way. Then yes, everytime the program starts, it allocates those strings in SRAM. I'm probably not using the appropriate terminology to explain myself.
You could create an array for each value that you want to use in strstr, and store those arrays in PROGMEM.
That is exactly my intention, now that I know it to be the root of the memory problem.
When I have 1KB+ of free memory, it's not that bad to allocate a larger buffer for NewSoftSerial.
On the other hand, although it may not seem to be a big deal right now, it most probably will be when I further develop the sketch. So a better software serial reading speed is still important (despite the fact that I don't know how to optimize the code posted above).
What I mean is that uploading the sketch will store the strings in program memory in some way.
No, it doesn't. During compilation, the compiler puts some stuff in SRAM and some stuff in program memory. For constant strings, they go in SRAM, and a pointer to the non-changing data in SRAM goes in program memory. The string itself never does.
Constant strings are not moved to SRAM at run time.
Sorry for that. I don't feel very comfortable denying something when I don't have full knowledge of what that something really is.
footswitch:
I'm probably not using the appropriate terminology to explain myself.
Which means I'm misusing the expression "program memory".
I'm trying to make the point here that the strings are placed somewhere when the Arduino is off.
I don't really know how it all works, but because you keep explaining it in different ways, I think I'm now a bit closer to reality.
The way I was seeing it before:
- There's a "program memory" where the compiled code is directly uploaded to;
- When the Arduino is powered on, it runs that compiled code.
And the way I see it now is:
- There's a memory that stores the code;
- When the Arduino is powered on, it compiles that code and decides where to allocate what;
- We can specifically tell the compiler to allocate certain things in certain places.
I may still be wrong or incorrect, so, if you really wish to keep correcting my assumptions, it'll be better for me to just stop talking about it, because I'm not actually contributing with anything new, nor correct, nor useful 
Thanks, PaulS.
OK, now that you know what it is, this is how you handle a string out the serial port.
char Dbuf[100];
strcpy_P(Dbuf,PSTR("********************long string that takes up space..********************"));
Serial.println(Dbuf);
// and then
strcpy_P(Dbuf,PSTR("********************yet another long string that takes up space..********************"));
Serial.println(Dbuf);
Notice how the same 100 bytes of Dbuf can be used over and over again? If you want to put variables in it, use two buffers:
char Dbuf[100];
char Dbuf2[100];
strcpy_P(Dbuf2,PSTR("******************** variable 1 = %d, variable2 = %d********************"));
sprintf(Dbuf,Dbuf2, variable1,variable2);
Serial.println(Dbuf);
And then use the two buffers over and over again. I've never been clear on exactly how the strings are allocated and where, etc. However, they really do seem to use twice as much space as they should unless you use PROGMEM to handle them. It really does appear that they are held in sram until runtime and then copied into ram. I can't think of another explanation why they use twice as much as one would think. However, that's another discussion. Just use the easy examples above and you can cut memory usage waaaay down and get on with your project.
edit: Oh, and notice that it's not strcpy(), it's strcpy_P. There's corresponding '_P' routines for most of the c string routines.
I may still be wrong or incorrect, so, if you really wish to keep correcting my assumptions, it'll be better for me to just stop talking about it, because I'm not actually contributing with anything new, nor correct, nor usefu
You are asking intelligent questions, though.
There are volatile memory areas on the Arduino, where stuff is temporary, and lost when the Arduino loses power or is reset.
There are non-volatile memory areas, where the hex file is stored. The hex file has several sections - data, code, and others. The data section contains the constant strings. The code section contains the machine code that gets executed. The data section occupies one part of the non-volatile memory. The code section occupies another part of non-volatile memory. The data section is limited in size, and is referred to as SRAM. The code section is also limited in size, but it's a much larger size.
The data section can be written to at run time. The code section can not. Using the PROGMEM directive causes the compile to put stuff that normally goes in the data section into the code section, instead.
Alternative means, therefore, are required to read that stuff.
draythomp...
Sorry for the late reply. It's just that after reading your post, I fainted. LOL
Ok, so what you're suggesting is I have to get some sleep (it's midnight here) and start reading on the subject tomorrow.
PROGMEM
strcpy_P()
PSTR()
...... Brain overflow.
Thanks for the advice
By the way, PaulS, I'm sold 
PaulS, that's not correct. There's three kinds of memory on the 328, Flash (32K), Sram (2K), and EEprom (1K). The flash can't be written by the program you run (thank the gods of electrons), the Sram and EEprom can be. The strings, something defined like: char thing[] = "Hello World"; can be written to, I've done it many times so it can't be in Flash. Also, the string will eat up space in the Flash memory, you can tell by the loading message from the IDE. Also, it eats Sram, you can tell by checking how much memory is left when the program is running.
Like I said, I don't know all the details, just how to work with it successfully and keep my Sram at the maximum so I can cram more work into the little guy.
Have a nice night footswitch, and when you get up and read this, it ain't hard, just copy the code and get on with life. Learn it in detail later.
Found it, this fits all my experiences:
Initial variable values and strings are copied out from program memory into RAM as part of the C startup routines, which execute before your main() function. Those startup routines give your globals their initial values, as well as ensure that all strings are inside RAM so they can be passed to your desired string handling routines. As more strings are added to your program, more data must be copied to RAM at startup and the more RAM is used up by holding static (unchanging) data.
So, the strings really do eat up twice the space you would think.
PaulS, that's not correct.
Specifically, what isn't? I'm willing to learn.
The strings, something defined like: char thing[] = "Hello World"; can be written to, I've done it many times so it can't be in Flash.
There's a big difference between a string variable and a constant string, like in Serial.print("ThisIsAString");
The string "ThisIsAString" is stored in SRAM, but it is not alterable, because you have no idea where in SRAM the string is.
The string "Hello World" is also stored in SRAM, and is alterable, because you do have an address where the data is stored.
I haven't got all the details yet, but I'm getting there slowly. The compiler does exactly what you said, separating things into data, code and so forth. This is ALL written into the Flash. That way, you kill the power and it's still there; remember, the entire Sram contents get clobbered when you power down. At startup, the startup code copies the data from Flash into Sram so you have strings and initialized variables. This means the various strings exist in the Flash and in the Sram after this process completes. This is what the various PROGMEM routines and macros do, get that stuff out of the Flash so you can use it as well as make sure it is part of the code segment so it doesn't get duplicated. A simple test of this can be constructed easily and is left as an exercise for the student (I hinted at this previously).
The data exists in both the Flash and the Sram after the initialization before setup() is called. This is actually pretty cool, but wasteful if you don't know about PROGMEM. But, by the time you run out of memory (excepting the String routines), you're ready to take this on anyway.
So, it all fits. Flash can't be written to by the program, the data is copied to Sram at start up hence the doubling of size, there's only 2K of Sram, so we run out pretty quickly if we're not careful especially if the data copied in takes up most of it as verbose debug strings tend to do.
More than I ever needed to know to blink a couple of leds.....
More than I ever needed to know to blink a couple of leds.....
Sure, but if you want to do more than blink a couple of LEDs, it's important (and fun) to know how the computer (of any size) works.