optimizing memory without Strings

I am trying to convert a sketch that used a lot of memory doing string manipulation using the string object to something more memory-friendly.

My assumption, based on what I’ve read, is that Strings and their methods are more dangerous to ram than char arrays, and cstring or homebrewed methods.

In the test sketch posted below there is a String and char verion of the same function with a helper freeRam() function found on this forum to report remaining ram. The String version actually performed better.

so my question is: How should I interpret this result in terms of finding strategies to make my larger sketch for optimized for memory?

My assumptions were that: 1) char is better than String generally, 2) PROGMEM could be useful, but using flash-allocated constants inside functions is very tricky, 3 passing arrays by reference to functions is not good for memory (I only read this somewhere);

Any help is greatly appreciated!

char statement[] = {"length of bench is ?Ft"};
char statement2[] = {"length of bench is ?Ft"};

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println(replacewithValue(statement, 44));
  Serial.println(replacewithValue2(statement, 55));
  delay(3000);
}


// String version:

String replacewithValue(String theString, int value) {
  String stringA = theString, stringB = theString;
  int start = 0, ending = 0;
  freeRam();
  for (int i = 0; i < theString.length(); i++) {
    if ( theString.charAt(i) == '?' ) {
      start = i;
      break;
    }
  }
  for (int i = theString.length(); i > 0; i--) {
    if ( theString.charAt(i) == '?' ) {
      ending = i + 1;
      break;
    }
  }
  freeRam();
  stringA.remove(start);
  stringB.remove(0, ending); freeRam();
  String rtn = stringA + value + stringB; freeRam();
  return rtn;
}

//char array version:

String replacewithValue2(char in[], int value) {
  int a, b = 0;
  char A[300]; char B[300]; char C[300];
  freeRam();
  for (int i = 0; i < strlen(in); i++) {
    if (in[i] != '?') {
      A[i] = in[i];
    } else {
      A[i] = '\0';
      a = i;
      break;
    }
  } freeRam();

  for (int i = a + 1; i < strlen(in); i++) {
    B[i - (a + 1)] = in[i];
    b++;
  }
  B[b] = '\0';
  freeRam();
  sprintf(C, "%s%d%s", A, value, B);
  freeRam();
  return C;

}

  //ram monitor function I found on this forum
void freeRam () {
  extern int __heap_start, *__brkval;
  int v;
  int rtn = (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
  Serial.println(rtn);
}

OUTPUT:
2220
2220
2220
2168
length of bench is 44Ft
1411
1411
1411
1401
length of bench is 55Ft

I changed the char buffers from 300, to 24, the minimum for size for the statement being passed. according to freeRam() this puts the two methods at a tie, with the String function having a big advantage of being flexible given it's input. I understand this is pretty limited test. I will repost with my full sketch after I edit it ti not contain some urls related to my job.

you want to insert a value in a “standard” string to display it?

As you always split up the string to insert a value, why not keep it as a split string or as format string

Check this

char prefix[] = "length of bench is";
char postfix[] = "Ft";

char formatFeet[] = "length of bench is %d Ft";
char output[24];

void setup()
{
  Serial.begin(9600);

  Serial.print(prefix);
  Serial.print(66);
  Serial.println(postfix);

  sprintf(output, formatFeet, 77);
  Serial.println(output);
}

void loop()
{

}

Or does the statement come from outside the sketch and you do not know where the ? char is?

yes it's a function from a larger sketch that substitutes a value for a '?' char in a statement. the returned string also gets manipulated in that larger sketch and added to another string, but the statement and other strings can change. I thought it would be a good function to test different memory saving strategies. I was just surprised that the char version wasn't any better, but maybe in the local scope of a function it doesn't matter or that an isolated test function can't simulate memory problems that would accumulate in a larger sketch.

OK than I try to rewrite

try this, should be faster too (for you to prove :wink:

char statement[] = "length of bench is ? Ft";
char statement2[] = {"length of bench is ?Ft"};

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.println(replacewithValue(statement, 44));
  Serial.println(replacewithValue2(statement, 55));
  Serial.println(replacewithValue3(statement, 66));
  delay(3000);
}


// String version:

String replacewithValue(String theString, int value) {
  String stringA = theString, stringB = theString;
  int start = 0, ending = 0;
  freeRam();
  for (int i = 0; i < theString.length(); i++) {
    if ( theString.charAt(i) == '?' ) {
      start = i;
      break;
    }
  }
  for (int i = theString.length(); i > 0; i--) {
    if ( theString.charAt(i) == '?' ) {
      ending = i + 1;
      break;
    }
  }
  freeRam();
  stringA.remove(start);
  stringB.remove(0, ending); freeRam();
  String rtn = stringA + value + stringB; freeRam();
  return rtn;
}

//char array version:
String replacewithValue2(char in[], int value) {
  int a, b = 0;
  char A[300]; char B[300]; char C[300];
  freeRam();
  for (int i = 0; i < strlen(in); i++) {
    if (in[i] != '?') {
      A[i] = in[i];
    } else {
      A[i] = '\0';
      a = i;
      break;
    }
  } freeRam();

  for (int i = a + 1; i < strlen(in); i++) {
    B[i - (a + 1)] = in[i];
    b++;
  }
  B[b] = '\0';
  freeRam();
  sprintf(C, "%s%d%s", A, value, B);
  freeRam();
  return C;

}



String replacewithValue3(char in[], int value)
{
  char C[32];   // max size resulting string
  char B[6];    // max size of number
  int ic = 0;   // index C
  int ii = 0;   // index In
  freeRam();
  while (in[ii] !=  '\0')
  {
    if (in[ii] == '?')
    {
      ii++; // skip ?
      itoa(value, B, 10);  // integer to ascii
      int ib = 0;
      while (B[ib] != '\0')
      {
        C[ic++] = B[ib++];
      }
    }
    else
    {
      C[ic++] = in[ii++];
    }
  }
  C[ic] = '\0';
  freeRam();
  return C;
}

//ram monitor function I found on this forum
void freeRam () {
  extern int __heap_start, *__brkval;
  int v;
  int rtn = (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval);
  Serial.println(rtn);
}

oraz:
My assumption, based on what I've read, is that Strings and their methods are more dangerous to ram than char arrays, and cstring or homebrewed methods.
...
My assumptions were that:

  1. char is better than String generally
  2. PROGMEM could be useful, but using flash-allocated constants inside functions is very tricky
  3. passing arrays by reference to functions is not good for memory (I only read this somewhere);

RAM is not endangered by any use, but you can kill it with overvoltage.
RAM does not care what its bits represent.
RAM usage and amout of contiguous RAM available after 20 days of operation is a different story.

  • its like saying "men are better than women", well, it depends
  • PROGMEM saves RAM usage and is only a little bit more complex to handle
  • passing arrays by value is costly as to RAM usage, passing them by reference is like passing an int
    So I think you should correct your assumptions.

No need to hurry if you are heading the wrong way.

The String version actually performed better.

That is probably one of the design goals of dynamic memory allocation.

@ robtillaart your code does perform a bit better than the string method. The difference is note huge though.

That is probably one of the design goals of dynamic memory allocation.

Yeah, I think I had some wrong impressions about ram space saving techniques based on forum threads that said it's a bad idea to use Strings when you can use char arrays. But in the scope of a local variable there doesn't seem to be much difference, and you have the benefit of not hoping your buffer length is small, yet safe. I tried using strlen(), but it was giving me problems. So I guess the string vs char points are more about strings created on a global or main loop scope.

oraz:
@ robtillaart your code does perform a bit better than the string method. The difference is note huge though.

Can you post the numbers?
Did you measure the difference in footprint?

Hi

My application at http://www.2wg.co.nz/ makes extensive use of Strings and very limited use of char arrays. I think it proves that you do not have to avoid Strings to build a robust fully featured Arduino application.

My application has been under development for two years and runs to well more than 10,000 lines of code. Over time I managed to work out how to use Strings and avoid the problems typically associated with them.

In general if you avoid global Strings (at least ones that you will be updating) and restricts String and String manipulation to within small local procedures you should be fine. String.reserve() is also recommended when you want to manipulate Strings.

Strings are dynamic memory objects, all be it objects that can be dynamically resized through use and can fragment your heap area. It is OK to use other dynamic objects such as ethernetclient and File objects - they too need to be used with care.

My application seems to run continuously without fragmenting free heap memory (which is the problem that Strings can create) - I have SRAM checking controls that keep a close watch on SRAM usage - the stack, the heap, free heap, free SRAM and the fixed data area.

Cheers

Catweazle NZ

CatweazleNZ:
you do not have to avoid Strings to build a robust fully featured Arduino application.

I think the main reason String got such a bad name was a bug in avr-libc 1.6.4’s malloc which caused a memory leak. This was fixed in Arduino IDE 1.0.5 I believe.

pert:
I think the main reason String got such a bad name was a bug in avr-libc 1.6.4's malloc which caused a memory leak. This was fixed in Arduino IDE 1.0.5 I believe.

I understand that however it is still easy to use Strings badly and that gives Strings a bad name because of the difficulties people get into when they do not understand the complex issue of heap fragmentation and its causes.

It is also not helpful (to the cause of Strings and the benefits they can bring to a project) that the overall message that Strings are bad is still often promoted on this forum.

Strings are typically easier to use for inexperienced programmers and they are a key target of the Arduino market.

And if Strings were indeed a bad idea they would be deprecated and eventually removed from the Arduino environment.

Catweazle NZ

I've been watching this thread with increasing discomfort:

...The String version actually performed better.
...That is probably one of the design goals of dynamic memory allocation.
...So I guess the string vs char points are more about strings created on a global or main loop scope.

Expecting a beginner to evaluate different approaches is ludicrous. robtillaart tried to point this out, in a gentle way, with replacewithValue3.

I think Catweazle has summed it up nicely:

  • it is still easy to use Strings badly
  • people do not understand the complex issues
  • Strings are typically easier to use for inexperienced programmersThese are the reasons I would never recommend String to a beginner. They would be better off in the long run by learning about arrays and why "strlen gave them problems". The beginner that starts with String and bangs out his project quickly, with great deal of satisfaction, eventually runs into The Problem.

Similarly, I would never expect to see an expert use String. They have (usually) learned that there are no net benefits to malloc/free (related thread here and countless other places on the intertubes). The "Easy" benefit is far outweighed by "Easy to use badly" and "Complex issues". In any embedded environment, predictability is paramount. Although it is possible to use Strings correctly (i.e., predictably), the expertise and care required to do so push (most) developers to other solutions. I especially encourage solutions that do not require a contiguous array of bytes for the entire string: Most of the OP's string should be in PROGMEM, not RAM.

  • And if Strings were indeed a bad idea they would be deprecatedOnly in a Star Trek Universe do bad ideas die. In this universe, Strings live on.

Cringingly,
/dev,
Man of La Malloc

@robtillaart

the output was
2217
2217
2217
2163
length of bench is 44 Ft
2215
2215
2215
2205
length of bench is 55 Ft
2275
2275
length of bench is 66 Ft

So the measurement byfreeRam() was 2275, a gain of 58 bytes.

EDIT: I misread, your code. the 6 byte value for char B makes sense...

people do not understand the complex issues

I really haven't found anybody that seemed to have an in depth knowledge of how the String functions actually perform dynamic allocation in the memory registers. If the String functions bit/byte shift the memory registers to defragment them when locations are freed up, then there is no end result fragmentation. It should be possible to shift the bytes in the registers to close the gaps, keeping the same byte count for the remaining allocated memory locations, and only update the indexes for the shifted allocations. By old school design, c-strings will probably have empty/wasted memory locations most of the time in the c-string memory locations. I guess the real answer lies at the machine level operations of the memory registries.

As far as I know when Strings are allocated they take the smallest free heap block available or expand the heap if there is not a large enough contiguous space. The same applies when you expand a String.

When any dynamic object is freed it become a block of free heap space, but if it was the last block on the heap then heap size is reduced instead.

Objects are not moved in the heap once allocated - this is what gives rise to free heap blocks within the heap when other objects or String areas are released.

When two adjoining blocks of free space occur they are merged into one larger free space block.

If you have no global dynamic objects or Strings and instantiate no dynamic objects or Strings in loop() then at the start of each iteration of loop() you will have a zero length heap and cannot suffer the slow death of a badly fragmented heap. Any dynamic objects or Strings created in procedures called by loop() should be fully released by the time program control gets back to the loop().

I do create ethernetclient objects that last for two seconds and multiple iterations of the loop(). But once I "stop" them and get back to loop() my heap is minimised (almost zero length) with no chunks of free space.

Anyway this is how I understand that it works.

Catweazle NZ

The CatweazleNZ story makes most sense to me because when I have a pointer to an object (or to inside an object) and the object is moved to fill a gap the pointer is pointing to something wrong.

You can see the malloc and free implementations here:

    <ARDUINO DIR>/hardware/arduino/cores/arduino/malloc.c

when I have a pointer to an object (or to inside an object) and the object is moved to fill a gap the pointer is pointing to something wrong.

Welcome, Pandora, to the Hopeless world of Dangling References, Garbage, Garbage Collection, and the numerous techniques for dealing with them. malloc, the all-giving memory function, inevitably opens this jar. In Java, reference management is part of the language. Ironically, Java code collections can be stored in .JAR files.

Pandora

Cheers,
/dev