Limits on size of String and using indexOf() ?

This is a basic programming question but here's the context: I am using a WiShield to send a GET to a remote server and process the returned data. The WiShield gets the data in chunks of maybe 50 characters at a time so several chunks have to get combined to get the entire response. Currently I'm doing that by concatenating each char with a String:

combineString += *(data);

where data is the incoming character. This creates a String of typically about 200 characters.

The problem is that when I use indexOf() to find something in this String, it fails with -1 returned. I did some tests and found that using indexOf() on Strings of more than about 160 characters, will fail, returning -1.

Why is this?

Also can anyone suggest perhaps a better approach to the problem? Thanks!

Which Arduino board / ATmel processor are you using?

david-x47: where data is the incoming character. This creates a String of typically about 200 characters.

Keep in mind that when a String object is concatenated, a new buffer the size of the original string plus 1 is created. Then the new result is copied into the original buffer. This means you are using 2x the storage required during the operation. So if you have 200 characters and add one, you at that moment, using 400 bytes.

david-x47: I did some tests and found that using indexOf() on Strings of more than about 160 characters, will fail, returning -1.

How did you generate these 160 characters? Was this by concatenating strings until you got 160 characters to search? If so, you might be running out of RAM which means the String isn't actually storing what you expected which is why indexOf is failing.

david-x47: Also can anyone suggest perhaps a better approach to the problem? Thanks!

If you know the maximum number of characters you are going to deal with, then just use a character array.

Thanks very much. My tests involved setting a string e.g.

String x = “11111111…”; // with 170 characters

However, I have decided instead to use a large char array, then construct a String from that array, and then indexOf() works fine.

Thanks!

However, I have decided instead to use a large char array, then construct a String from that array, and then indexOf() works fine.

Good grief. Some people just will not do things the smart way.

Why don't you look at what indexOf() does, and do that to your character array, instead of wasting all the resources that String wastes?

The String class is #1 on my list of things not to use with Arduino, see http://miscsolutions.wordpress.com/2011/10/16/five-things-i-never-use-in-arduino-projects/. Writing a program for an embedded processor with very limited RAM and limited performance is not the same as writing an application to run on a PC.

Many thanks to James C4S for a helpful and polite response!

As for PaulS,

Good grief. Some people just will not do things the smart way.

I'm not accustomed to essentially being called stupid for asking what I believe was a thoughtfully-composed question backed by research and experimentation, and qualified as a 'beginner' question. Do you talk to people like this in person? If you tire of questions that don't meet your personal hygienic definition of what a question should be, then you should have stopped reading after I qualified my question right off the bat as being in the 'newbie' category. A question by definition means that I am in some respects ignorant of something, or else I wouldn't be asking for help.

I've seen other arrogant responses like this from him and am surprised that it seems few people chime in when this happens. You've got an otherwise good community here, and if one of your most prolific posters makes a habit of frequently posting snide responses, that's not doing this community much good, since many will not come back, especially since Arduino is considered to be a good starting place for beginners.

Perhaps the moderator would consider adding some more modern forum features (see the outstanding StackOverflow.com for example) that allow posts to be rated 'up' or 'down', and that in essence penalize bad behavior, rather than rewarding it for more Post counts (as if those meant anything anyway?). I'd rather have a polite, decent response, than get slammed and get whatever follows.

It was pointed out that the String class was the source of your problem. You insist on using it anyway. OK, fine, it's your code and you can do what you like with your code.

But, there is no way that I can consider it smart.

People have written extremely complex string (lower case s) manipulation code using the same string handling functions that the String class (upper case S) wraps, without the memory manglement issues that the String class presents.

It is not that difficult to find the source code for the String class and to locate the indexOf() function in the class, to see how the class performs the task of locating the first occurrence of a particular character in an array, if it isn't obvious which string method to use.

The String class should, at best, be a reference, not a tool that actually gets used. At least not for anything more than 10 characters in length.

If you are already storing the character data in a character array, using the String class so you can use the indexOf() function is a crutch.

Perhaps the moderator would consider adding some more modern forum features (see the outstanding StackOverflow.com for example) that allow posts to be rated ‘up’ or ‘down’, and that in essence penalize bad behavior, rather than rewarding it for more Post counts (as if those meant anything anyway?). I’d rather have a polite, decent response, than get slammed and get whatever follows.

The board owners did that when this new board was started. Was a total disastor. If you stay around the board for a while, you will notice members that appear to have issues with nubie questions. People are what they are.

To give credit where it’s due, PaulS has a high post count because he answers many of the questions put to the forum. Sometimes he might be a bit terse, but I imagine after 1000 posts along the lines of “I am having problems with my code (code not shown) and am getting an error message (error message not shown)” one’s patience might wear a bit thin.

Sometimes I see questions I don’t bother attempting to answer because the poster clearly has not made the slightest attempt to help themselves (eg. trying the examples, Google, browsing the Arduino web site). Often enough though, PaulS will answer with at least a hint about where to look, or a suggestion about how to make the request more understandable.

Hello all,
I know it is an old post, but I have not seen a constructive answer to the problem. Here is what I suggest.

As for your actual problem (String.indexOf() > 160 - fail):
Either string is empty, or what you are searching for is not there.

You are most likely testing by printing the contents of the string to serial terminal.
Make sure to call Serial.flush() to force printing a complete debug info.

Another possible reason (mentioned above) - you have reached the limit of your RAM and appending some new characters to your string without checking the length of the string. String::concat() returns 0 if the concatenation failed. It is not very elegant but more safe to do that.

As for the “better approach” to do what you are doing - you approach is fine. The problem is - the approach is lucking some predictability. That is why you have gotten so many negative responses.

I would suggest the following:

#define MAX_GET_RESPONSE_BYTES 200 /* Adjust the size to your needs */
String responseString;
if( responseString.reserve( MAX_GET_RESPONSE_BYTES) == 0) {
   Serial.println( "ERROR: Failed to allocate GET response buffer");
}
else {
   /* TODO: Use the string for your GET response processing */
}

The benefit of this approach is that you have all the features String offers and still can handle the “out of memory” situation.

Few things to be aware of:

  • Play around with MAX_GET_RESPONSE_BYTES and make sure to figure out the right size
  • If you allocate responseString on stack (within some function) the buffer will be gone after you leave the function - don’t return references or pointers to it
  • If you allocate responseString globaly - it will hold all this MAX_GET_RESPONSE_BYTES for a life time of the program - make sure that is what you need.

Source: /arduino-1.8.5/hardware/arduino/avr/cores/arduino/WString.cpp

Regards the etiquette thing, I actually see that the OP shrugged off the sagely advice in the very first post. The advice was “smart”. So saying that what the OP intends to do is “not smart” is just calling a shovel a shovel.