Go Down

Topic: optimizing memory without Strings (Read 9149 times) previous topic - next topic

oraz

I am trying to convert a sketch that used a lot of memory doing string manipulation using the string object to something more memory-friendly.

My assumption, based on what I've read, is that Strings and their methods are more dangerous to ram than char arrays, and cstring or homebrewed methods.

In the test sketch posted below there is a String and char[] verion of the same function with a helper freeRam() function found on this forum to report remaining ram. The String version actually performed better.

so my question is: How should I interpret this result in terms of finding strategies to make my larger sketch for optimized for memory?

My assumptions were that: 1) char[] is better than String generally, 2) PROGMEM could be useful, but using flash-allocated constants inside functions is very tricky, 3 passing arrays by reference to functions is not good for memory (I only read this somewhere);

Any help is greatly appreciated!
Code: [Select]


char statement[] = {"length of bench is ?Ft"};
char statement2[] = {"length of bench is ?Ft"};

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println(replacewithValue(statement, 44));
  Serial.println(replacewithValue2(statement, 55));
  delay(3000);
}


// String version:

String replacewithValue(String theString, int value) {
  String stringA = theString, stringB = theString;
  int start = 0, ending = 0;
  freeRam();
  for (int i = 0; i < theString.length(); i++) {
    if ( theString.charAt(i) == '?' ) {
      start = i;
      break;
    }
  }
  for (int i = theString.length(); i > 0; i--) {
    if ( theString.charAt(i) == '?' ) {
      ending = i + 1;
      break;
    }
  }
  freeRam();
  stringA.remove(start);
  stringB.remove(0, ending); freeRam();
  String rtn = stringA + value + stringB; freeRam();
  return rtn;
}

//char array version:

String replacewithValue2(char in[], int value) {
  int a, b = 0;
  char A[300]; char B[300]; char C[300];
  freeRam();
  for (int i = 0; i < strlen(in); i++) {
    if (in[i] != '?') {
      A[i] = in[i];
    } else {
      A[i] = '\0';
      a = i;
      break;
    }
  } freeRam();

  for (int i = a + 1; i < strlen(in); i++) {
    B[i - (a + 1)] = in[i];
    b++;
  }
  B[b] = '\0';
  freeRam();
  sprintf(C, "%s%d%s", A, value, B);
  freeRam();
  return C;

}

  //ram monitor function I found on this forum
void freeRam () {
  extern int __heap_start, *__brkval;
  int v;
  int rtn = (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
  Serial.println(rtn);
}




OUTPUT:
2220
2220
2220
2168
length of bench is 44Ft
1411
1411
1411
1401
length of bench is 55Ft

oraz

I changed the char[] buffers from 300, to 24, the minimum for size for the statement being passed. according to freeRam() this puts the two methods at a tie, with the String function having a big advantage of being flexible given it's input. I understand this is pretty limited test. I will repost with my full sketch after I edit it ti not contain some urls related to my job.

robtillaart

#2
Oct 17, 2015, 09:18 am Last Edit: Oct 17, 2015, 09:20 am by robtillaart
you want to insert a value in a "standard" string to display it?

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

robtillaart

As you always split up the string to insert a value, why not keep it as a split string or as format string

Check this
Code: [Select]

char prefix[] = "length of bench is";
char postfix[] = "Ft";

char formatFeet[] = "length of bench is %d Ft";
char output[24];

void setup()
{
  Serial.begin(9600);

  Serial.print(prefix);
  Serial.print(66);
  Serial.println(postfix);

  sprintf(output, formatFeet, 77);
  Serial.println(output);
}

void loop()
{

}


Or does the statement[] come from outside the sketch and you do not know where the ? char is?

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

oraz

yes it's a function from a larger sketch that substitutes a value for a '?' char in a statement. the returned string also gets manipulated in that larger sketch and added to another string, but the statement and other strings can change. I thought it would be a good function to test different memory saving strategies. I was just surprised that the char version wasn't any better, but maybe in the local scope of a function it doesn't matter or that an isolated test function can't simulate memory problems that would accumulate in a larger sketch.

robtillaart

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

robtillaart

#6
Oct 17, 2015, 10:53 am Last Edit: Oct 17, 2015, 10:53 am by robtillaart
try this, should be faster too (for you to prove ;)
Code: [Select]
char statement[] = "length of bench is ? Ft";
char statement2[] = {"length of bench is ?Ft"};

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println(replacewithValue(statement, 44));
  Serial.println(replacewithValue2(statement, 55));
  Serial.println(replacewithValue3(statement, 66));
  delay(3000);
}


// String version:

String replacewithValue(String theString, int value) {
  String stringA = theString, stringB = theString;
  int start = 0, ending = 0;
  freeRam();
  for (int i = 0; i < theString.length(); i++) {
    if ( theString.charAt(i) == '?' ) {
      start = i;
      break;
    }
  }
  for (int i = theString.length(); i > 0; i--) {
    if ( theString.charAt(i) == '?' ) {
      ending = i + 1;
      break;
    }
  }
  freeRam();
  stringA.remove(start);
  stringB.remove(0, ending); freeRam();
  String rtn = stringA + value + stringB; freeRam();
  return rtn;
}

//char array version:
String replacewithValue2(char in[], int value) {
  int a, b = 0;
  char A[300]; char B[300]; char C[300];
  freeRam();
  for (int i = 0; i < strlen(in); i++) {
    if (in[i] != '?') {
      A[i] = in[i];
    } else {
      A[i] = '\0';
      a = i;
      break;
    }
  } freeRam();

  for (int i = a + 1; i < strlen(in); i++) {
    B[i - (a + 1)] = in[i];
    b++;
  }
  B[b] = '\0';
  freeRam();
  sprintf(C, "%s%d%s", A, value, B);
  freeRam();
  return C;

}



String replacewithValue3(char in[], int value)
{
  char C[32];   // max size resulting string
  char B[6];    // max size of number
  int ic = 0;   // index C
  int ii = 0;   // index In
  freeRam();
  while (in[ii] !=  '\0')
  {
    if (in[ii] == '?')
    {
      ii++; // skip ?
      itoa(value, B, 10);  // integer to ascii
      int ib = 0;
      while (B[ib] != '\0')
      {
        C[ic++] = B[ib++];
      }
    }
    else
    {
      C[ic++] = in[ii++];
    }
  }
  C[ic] = '\0';
  freeRam();
  return C;
}

//ram monitor function I found on this forum
void freeRam () {
  extern int __heap_start, *__brkval;
  int v;
  int rtn = (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
  Serial.println(rtn);
}
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Whandall

My assumption, based on what I've read, is that Strings and their methods are more dangerous to ram than char arrays, and cstring or homebrewed methods.
...
My assumptions were that:
1) char[] is better than String generally
2) PROGMEM could be useful, but using flash-allocated constants inside functions is very tricky
3) passing arrays by reference to functions is not good for memory (I only read this somewhere);
RAM is not endangered by any use, but you can kill it with overvoltage.
RAM does not care what its bits represent.
RAM usage and amout of contiguous RAM available after 20 days of operation is a different story.

  • its like saying "men are better than women", well, it depends
  • PROGMEM saves RAM usage and is only a little bit more complex to handle
  • passing arrays by value is costly as to RAM usage, passing them by reference is like passing an int
So I think you should correct your assumptions.

No need to hurry if you are heading the wrong way.
Ah, this is obviously some strange usage of the word 'safe' that I wasn't previously aware of. (D.Adams)

zoomkat

Quote
The String version actually performed better.
That is probably one of the design goals of dynamic memory allocation.
Google forum search: Use Google Search box in upper right side of this page.
Why I like my 2005 Rio Yellow Honda S2000  https://www.youtube.com/watch?v=pWjMvrkUqX0

oraz

#9
Oct 17, 2015, 11:48 pm Last Edit: Oct 19, 2015, 12:16 am by oraz
@ robtillaart your code does perform a bit better than the string method. The difference is note huge though.

Quote
That is probably one of the design goals of dynamic memory allocation.
Yeah, I think I had some wrong impressions about ram space saving techniques based on forum threads that said it's a bad idea to use Strings when you can use char arrays. But in the scope of a local variable there doesn't seem to be much difference, and you have the benefit of not hoping your buffer length is small, yet safe. I tried using strlen(), but it was giving me problems. So I guess the string vs char[] points are more about strings created on a global or main loop scope.

robtillaart

@ robtillaart your code does perform a bit better than the string method. The difference is note huge though.
Can you post the numbers?
Did you measure the difference in footprint?




Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

CatweazleNZ

#11
Oct 18, 2015, 11:51 am Last Edit: Oct 18, 2015, 11:57 am by CatweazleNZ
Hi

My application at http://www.2wg.co.nz/ makes extensive use of Strings and very limited use of char arrays. I think it proves that you do not have to avoid Strings to build a robust fully featured Arduino application.

My application has been under development for two years and runs to well more than 10,000 lines of code. Over time I managed to work out how to use Strings and avoid the problems typically associated with them.

In general if you avoid global Strings (at least ones that you will be updating) and restricts String and String manipulation to within small local procedures you should be fine. String.reserve() is also recommended when you want to manipulate Strings.

Strings are dynamic memory objects, all be it objects that can be dynamically resized through use and can fragment your heap area. It is OK to use other dynamic objects such as ethernetclient and File objects - they too need to be used with care.

My application seems to run continuously without fragmenting free heap memory (which is the problem that Strings can create) - I have SRAM checking controls that keep a close watch on SRAM usage - the stack, the heap, free heap, free SRAM and the fixed data area.

Cheers

Catweazle NZ

pert

you do not have to avoid Strings to build a robust fully featured Arduino application.
I think the main reason String got such a bad name was a bug in avr-libc 1.6.4's malloc which caused a memory leak. This was fixed in Arduino IDE 1.0.5 I believe.

CatweazleNZ

I think the main reason String got such a bad name was a bug in avr-libc 1.6.4's malloc which caused a memory leak. This was fixed in Arduino IDE 1.0.5 I believe.
I understand that however it is still easy to use Strings badly and that gives Strings a bad name because of the difficulties people get into when they do not understand the complex issue of heap fragmentation and its causes.

It is also not helpful (to the cause of Strings and the benefits they can bring to a project) that the overall message that Strings are bad is still often promoted on this forum.

Strings are typically easier to use for inexperienced programmers and they are a key target of the Arduino market.

And if Strings were indeed a bad idea they would be deprecated and eventually removed from the Arduino environment.

Catweazle NZ

-dev

I've been watching this thread with increasing discomfort:

Quote
...The String version actually performed better.
...That is probably one of the design goals of dynamic memory allocation.
...So I guess the string vs char[] points are more about strings created on a global or main loop scope.


Expecting a beginner to evaluate different approaches is ludicrous.  robtillaart tried to point this out, in a gentle way, with replacewithValue3.

I think Catweazle has summed it up nicely:

  • it is still easy to use Strings badly
  • people do not understand the complex issues
  • Strings are typically easier to use for inexperienced programmers
These are the reasons I would never recommend String to a beginner.  They would be better off in the long run by learning about arrays and why "strlen gave them problems".  The beginner that starts with String and bangs out his project quickly, with great deal of satisfaction,  eventually runs into The Problem.

Similarly, I would never expect to see an expert use String.  They have (usually) learned that there are no net benefits to malloc/free (related thread here and countless other places on the intertubes).  The "Easy" benefit is far outweighed by "Easy to use badly" and "Complex issues".  In any embedded environment, predictability is paramount.  Although it is possible to use Strings correctly (i.e., predictably), the expertise and care required to do so push (most) developers to other solutions.   I especially encourage solutions that do not require a contiguous array of bytes for the entire string: Most of the OP's string should be in PROGMEM, not RAM.

  • And if Strings were indeed a bad idea they would be deprecated
Only in a Star Trek Universe do bad ideas die.  In this universe, Strings live on.

Cringingly,
  /dev,
    Man of La Malloc
Really, I used to be /dev.  :(

Go Up