A Safe Alternative to using Strings in Arduino

Projects Programming
21

Think about it this way - wouldn't it be great if the normal C++ cstring functions were extended to include the capabilities of this library?

I see that a bit like the discussion between using digitalWrite() versus PORT manipulation.

The cString functions were written with efficiency in mind, ie if you know what you are doing there is no need for the "belt and suspenders" to double check all will be fine every time you do something. What you get is efficient code - possibly failing in miserable way if you have not been careful.

But given a new, non standard, library to learn versus learning about what's available to do those checks and cString operations my vote would go to teach the standard stuff...

this class is half-baked in my opinion because it does not offer the flexibility of expanding the underlying buffer (with all the issues attached) which is one of the value of the String class for newbies. ==> So you still need to catch overflows and handle buffer size anyway as you would have done in the first place with the standard functions and plain cStrings

The only bonus is it does not crash the Arduino miserably if you try to overflow, but I'm pretty sure in most cases the rest of the code then would not know how to handle a buffer that does not include what was expected.... so the crash might be just a few lines later..

22

I am still in favor of creating a good tutorial here which explains the 'pitfalls' of "Strings' , reason behind these pitfalls, possible solutions and examples of how to avoid those pitfalls and on how to properly use c-strings. But admittedly i am not much of a tutorial writer, i am more someone that takes a person by the hand and guides them through a process.

That's a bit like saying multiplication is a lazy way to do addition.

That is not really what i meant, i was more intended in the way that as a lazy option compared to learning how to work with c-strings to fix a sketch that may have heap-fragmentation issues (and it is difficult to get support for until those possible issues are gone)

23

I am still in favor of creating a good tutorial here which explains the 'pitfalls' of "Strings' , reason behind these pitfalls, possible solutions and examples of how to avoid those pitfalls

I often reference the Evils of Strings page when discussing the use of Strings with Arduino. I try not to say, "do not use Strings", but point the member to that article so that they can make up their own minds.

I am not sure about this new library. Having learned to use c_strings I don't feel the need to learn new methods.
But that's me.

24

J-M-L:
this class is half-baked in my opinion because ...

That's a very different type of argument against the use of the SafeString library and one which I would find much more persuasive. However I wonder how much work would be required to fix the problems you see?

Most of the other arguments against it come down to "I don't like it because it's different"

...R

25

Let's be real. Cstrings are tricky for a new programmer. Let me be even more bold - they're a little awkward. They were designed for programmers that "really know what they're doing". They were designed to make text work on a 12 bit system with 4k RAM and 10 inch floppy disks, or if you were really lucky, a 1 megabyte hard drive. They can be learned, but a good wrapper interface isn't really such an evil thing, if it's well written and documented. It looks good to me, although I probably won't use it. My text handling requirements are almost always extremely simple.

I suppose we should embrace anything that is good software design - we preach it all the time here. Abstraction is not evil, it's a miracle tool! The point is to avoid unnecessary abstraction - a little hard to qualify, but I guess it would be abstraction that leads to increased complexity vs. increased simplicity. In a global sense, you have to consider the cost of situations where you need to work behind the abstractions and you have no experience or knowledge of how to do that, because the inner workings were hidden from view. But over the long period of time during which a programmer is made, many opportunities will occur for learning aspects of the system at a very low, mid, and high level - for example, port writes, boolean logic, functions, classes from low to high. It's hard to avoid in this environment which is why it's such a good learning platform.

In terms of just getting the average project going, learning all that standard library stuff is usually very peripheral to what people want to focus on. So if they have a tool to make it easier, what harm is it?

26

Robin2:
That's a very different type of argument against the use of the SafeString library and one which I would find much more persuasive. However I wonder how much work would be required to fix the problems you see?

Most of the other arguments against it come down to "I don't like it because it's different"

...R

I'm not against the work that has been done. it's a great effort to address a recurring issue and it's true that cStrings are tricky for new programmers and a cause of major bugs.

On the positive it can have an educational value. You get reporting about things you forgot to check and given how hard it is for newbies to debug that could help pinpoint their issue.

It might have a negative consequence though in that newbies will just think "I need a bigger buffer"... Questioning how big, or why, or do I need it etc might not come naturally.

I'm looking at this from the other angle - which is "if I were a newbie, would I want to invest time to learn this class, which is somewhat unique to the Arduino world, or isn't it better to invest my time in understanding how to deal with cStrings and have that knowledge I can apply everywhere?" or the angle of helper on this forum or in computer club : "would I rather guide someone to understand cStrings or this class?"

The reason this class is "safe" is that it does not address the hard problem of dynamic expansion or temporary objects or returning a String from a method call — which I believe is what newbie expect (without really understanding).

Newbie want to not have to worry about code like

...
  int x, y; 
  x = ...
  y = ...
  String cmd = "do this with x = " + String(x) + "and y = " + String(y);
  return cmd;
}

it should just work.... And it does most of the time for a newbie program that runs for a few minutes as an exercise.

Fixing this and claiming you are staying safe is not trivial on a UNO...and comes with.... strings attached :slight_smile:

It's also safe because it does not crash if you try to overflow a buffer. But as I'm saying above what good does it bring to me if I'm left with strings that do not contain what I expected ? A message in the console is nice but if my SQL request is incomplete or a ASCII command truncated when I send it because my buffer was too small, I still have a buggy program.

That means in the end I still need to check bounds and size when I do any method call which is exactly what you do with cString and standard C programing style... (and where are the functions to do this.. text in the console does not help there ??)

So all things considered, I won't recommend that class to anyone - I would still encourage folks to use the standard cStrings and associated libraries if they want to write good code for microcontrollers.

Side note: (because it's irritating and I never liked marketing BS) I also would argue that many of the claims the author makes about being safe are marketing claims and just anchored in usual best practice (passing by reference, not blocking the main thread, ...) or lack of capabilities described above dressed up as good things because they are hard to solve problems and create the issues that exist with the String class...

27

I often reference the Evils of Strings page when discussing the use of Strings with Arduino

That is a great article.

Newbie want to not have to worry about code like
Code: [Select]
...
int x, y;
x = ...
y = ...
String cmd = "do this with x = " + String(x) + "and y = " + String(y);
return cmd;
}
it should just work.... And it does most of the time for a newbie program that runs for a few minutes as an exercise.

And it is totally safe to do it like this depending on whatever else is going on within your program. As long as all fragments are destroyed at some point, even if it is only at the end of loop(). Saying that, it means also that global declarations on the heap should not expand ! beyond their declared size. So manually a programmer should divide the heap into 2 sections. 1 Section that is always in use and has global declarations with a fixed maximum size. This includes for instance Neopixel objects that declare a buffer on the heap. And global Strings, for which you can reserve() space (first thing in setup() ) as one would do with c-strings. Sometimes libraries use the heap, and as long as they do clean up after themselves that should not be an issue, but for instance the neopixel.h library allows modification of the buffer size, and if you do that, you may fragment the heap, if you are also using the heap for other things like Strings, so care needs to be taken.
And a second second which is free to muddle up and fragment, but is completely empty at some stage of the program flow, and then can be re-used there after. On my Hard-drive i do the same, i have a drive where i store data that i keep for a long time, like music, pictures etc, an area from which i don't delete much and which i don't de-fragment very often (if at all)

They were designed to make text work on a 12 bit system with 4k RAM and 10 inch floppy disks,

that is still more than the UNO most people start of with. Mind you memory management has been ingrained in my as a child using a 1KB RAM machine, without non-volatile memory available to the programmer (man that 16KB upgrade was a big help, if only it hadn't been soooo un-reliable) Sharing the whole of memory between the program, the variables and the screen was just something to get your head around once, and becomes second nature.

So all things considered, I won't recommend that class to anyone

neither will I. Memory management is a fundamental part of programming, and it can be done safely using 'Strings' and 'c-strings' and adding a 3rd option really seems like a complication i don't want to be bothered about.

28

J-M-L:
I'm looking at this from the other angle - which is "if I were a newbie, would I want to invest time to learn this class, which is somewhat unique to the Arduino world, or isn't it better to invest my time in understanding how to deal with cStrings and have that knowledge I can apply everywhere?" or the angle of helper on this forum or in computer club : "would I rather guide someone to understand cStrings or this class?"

I think there are two very different types of newbie. Some of them will see an Arduino as a stepping stone to more general computing but my guess is that most of them just want to get their project to work and will never take their knowledge outside the Arduino realm. It is the latter who will benefit from this library IMHO - just as they benefit from analogWrite() etc. And I suspect anyone who is going to succeed in the wider world will be savvy enough to explore and learn more for themselves.

Side note: (because it's irritating and I never liked marketing BS) I also would argue that many of the claims the author makes about being safe are marketing claims and just anchored in usual best practice (passing by reference, not blocking the main thread, ...) or lack of capabilities described above dressed up as good things because they are hard to solve problems and create the issues that exist with the String class...

That does not seem to me a sufficient reason not to recommend the library. You do not seem to be saying the library fails to do what it claims.

I will admit to being in a strange position on this because I can't imagine doing an Arduino project that needs the capabilities if this library. I prefer to do textual stuff on my PC with Python. However I see lots of people doing things on Arduinos that do involve text.

...R

29

Well let me Think a bit aloud

I have no use for it Personally

When I teach stuff (to kids interested in going beyond their homework) i would go for the standard C libraries as this is what makes sense

If someone has a failing code with the String class I would probably first recommend using reserve and avoid the dynamic allocations (split concats, pass as reference,...) or move to CString.

Most newbie code I’ve seen would run just fine with the String class for their homework.

What Adafruit and the likes say is not to use it in a library. Newbies don’t publish libraries

The claims are not wrong but I read them as a lot of lipstick on what the class does and why it does not do X or Y - which is somewhat fishy Because the real reason is that you need dynamic allocation to do so and there is lots of value for newbies in this simplicity

That being said

There are bugs in the String class that needs fixing. He is right about those.

Those who would want a bit of handholding during development could see some debug benefit. Agreed. Would they need the training wheels all the time though - probably not and would have they been better off investing their time in cString in the first place - probably...as they will need some work to rewrite their code to add the checks to make it robust.

But hey that’s really a personal preference and it’s a free world - so those interested in using it should do so!

30

I'm beginning to lose interest in this library simply because there has been no participation in the discussion by the OP.

If he has no interest in promoting his own creation .....

...R

31

well he has been posting the same story to instructables, his web site, gitmemory, GitHub and here in the arduino forum. So there is an attempt to make it visible.

but agree it's a miss to not have the author's view.

32

J-M-L:
well he has been posting the same story to instructables, his web site, gitmemory, GitHub and here in the arduino forum. So there is an attempt to make it visible.

It's one thing to create something and "throw it over the wall so the kids can kick it about"

But the real value is in follow-up support.

I suspect the OP would have got more support if he had exposed the project at an earlier stage in its development so that he could more easily take account of comments and suggestions.

...R

33

agreed

According to the GitHub, it seems this is a company, not an individual, owning this.

Note, this is NOT my work, I am simply hosting it for easy access. The original code belongs to Forward Computing and Control Pty. Ltd.

So may be more complicated to make comments on behalf of the company.

Side note - this is what I've in mind when I hear safe strings

34

It’d be fun to run the various Arduino String libraries on a desktop with an instrumented set of malloc functions, and analyze in detail jus where it falls apart...

35

J-M-L:
According to the GitHub, it seems this is a company, not an individual, owning this

Ah, well ..... it was fun while it lasted :slight_smile:

...R

36

It'd be fun to run the various Arduino String libraries on a desktop with an instrumented set of malloc functions, and analyze in detail jus where it falls apart...

While i was at work today i considered the pros and cons of 'Strings' and what came to mind was actually that it could be really nice to have a way to de-fragment the heap, somewhere within the program cycle. Basically just a function that can be called, normally at the start of loop() when the heap is mostly empty, but in general anywhere.
It would take some time to move the data around, but in many cases that would be totally OK, with any newbie, and probably with most others in many projects, and definitely with any of the supporters here on the forum. I am just not really sure how hard it would be to do that, but it can be that difficult.

37

Moving things around requires changing pointers in your variables

If I allocate through malloc some buffer and keep a computed pointer in the middle of this buffer How do you know ? C or C++ let you do what you want with memory and pointers

unless you have a virtual memory architecture providing indirection its a pretty hard problem to solve.

That’s what modern OS provide but you need memory for this... catch 22

38

Even if you could implement garbage collection and heap restructuring, it can take a while and that while varies. Which means that your ability to control timing would be gone. Things like LED strips could fail intermittently. C strings all the way I think.

39

a way to de-fragment the heap

All the allocated memory has to stay in the same place, and the free() function already supports aggregation of free'ed memory. :frowning:
And we haven't actually proven that fragmentation is to blame.

Even if you could implement garbage collection and heap restructuring, it can take a while and that while varies.

Nevertheless, garbage collection has been implemented on systems with comparable CPU speed and much more RAM... If it was triggered at user request, it could be usable.

C strings all the way

You could wrap a set of C++ overloaded operators around C strings. Oh wait; that's essentially what this new library does...

40

westfw:
You could wrap a set of C++ overloaded operators around C strings. Oh wait; that's essentially what this new library does...

Or... perhaps it's also already been done previously...