String corrupts the heap and crashes

[nickgammon]

The bug report has a link to this thread:

http://arduino.cc/forum/index.php/topic,95914

On page 3 of that thread I posted the fixed code for 'free', showed how to temporarily incorporate it in the current IDE, and showed that it fixed the problem.

[westfw]

I just thought I'd point out (now that I'm back from vacation and have a real keyboard) that there is a significant logistic problem here.  The Arduino team doesn't "support" the C compiler or library provided with the Arduino environment.  They don't even "pick and choose" particular versions of gcc/avr-libc/etc.  What is included in avr-gcc is a compiler package that someone else has put together (WinAVR for Windows, CrossPack for Mac.)  And yes, it's a fairly old set of utilities, mainly because it's been a while since the WinAVR folks have put together a new package (Crosspack was based on WinAVR, so the Mac and PC versions of the Ardunio code are supposed to match.)  It's also been a while since a newer C compiler hasn't had "known bugs" that were critically unacceptable to Arduino (understand that Arduino is one of the few "consumers" of the gcc C++ support for AVRs.)

Applying a patch to avr-libc's free() means that the Arduino team would need to maintain their own version of the gcc tools distribution.  Updating the tools in the absence of a pre-packaged set (ie upgrading avr-libc without getting a new winAVR) is nearly as bad (equivalent to maintaining a WinAVR package.)  All of this is *HARD*.  Even upgrading to a new WinAVR (which is supposed to come out "real soon now") is likely to be a testing and compatibility nightmare.  (for instance, the gcc folk have decided that the way that avr-libc implements most of the "pgmspace" library )for storing constants in flash memory instead of ram) is "wrong and has never been supported."  There's a replacement scheme, but it IS likely to be different.

It's a tough problem.  It's very common for large projects (commercial and otherwise) to be several years behind "current" on compiler tools, just because upgrading the compiler is such a pain in the neck...

(And I noticed that the "official" avr-libc response to the bug in free was a complete rewrite of the memory allocator.  I can't tell you how little confidence that inspires :-( )

[nickgammon]

OK, well my work-around solution of adding "myfree" to somewhere that is likely to be included in a normal compile (eg. wiring.c) and then defining free to be myfree, might be the least intrusive.

This avoids any issues of what other changes might be made to the toolchain, it simply replaces one function with another that fixes a particular bug. And the linker should keep the code sizes the same, by using the one that is actually invoked.

I don't particularly like it for various reasons, but it is better than having a version of free that basically can't be used with confidence.

[pjrc]

Oh come on Bill, this really isn't so hard.  If it were, how would you explain String working for the last 2 years on Teensy?

[fat16lib]

The free bug() explains another problem I have searched for in SD.h. 

Often users of the SD.h library open and close files frequently.  Some of the SD.h examples open a file, write a line, and close the file each time through loop.  These programs sometimes crash in strange ways.

SD.h allocates and frees memory to for the SdFat file object and file name.  I got really close but never suspected that free could be the cause.

I did a search for free in 1.01 and found these lines in the core and libraries:

D:\arduino-1.0.1\hardware\arduino\cores\arduino\new.cpp
     10: free(ptr);
D:\arduino-1.0.1\hardware\arduino\cores\arduino\WString.cpp
    104: free(buffer);
    121: if (buffer) free(buffer);
    172: free(buffer);
D:\arduino-1.0.1\libraries\Firmata\Firmata.cpp
    391: free(tmpArray);
D:\arduino-1.0.1\libraries\SD\File.cpp
    134: free(_file);

Looks like String and SD.h are the big problems.  I suspect new() and Firmata are used very little.

[nickgammon]

new() was added in 1.0 of the IDE. I doubt people use it much. You need it for the STL, but anyone knowledgeable enough to use that would hold off until memory allocation is fixed.

Come to think of it, my Big Number port uses malloc and free heavily. I suppose they can't be considered reliable right now either.

[Bill52]

@ Paul Stoffregen
You are the most knowledgable as far as I can see.

Would it be too much to ask you to release an istaller for arduino boards?
If it has to be re-compiled, so be it. Give us a webpage with the instructions!

I am NOT proposing a breakaway branch!
Most users will be happy with the 'official' release as they will never use malloc/new or build anything neer a mission critical application.
But some of us do. It would be - at least a partial - solution.

What do you think?

Will

[m_ri]

i'm agree with bill52..thank you for your time!!

[pjrc]

Would it be too much to ask you to release an istaller for arduino boards?

That would be asking PJRC to support non-PJRC products.  The Teensyduino installer is designed to never modify Arduino for non-PJRC boards.

At the moment, I'm working on a new Teensy product, so I don't have time for extra stuff lately.

It would be - at least a partial - solution.

Isn't simply copying the malloc.c I provided (which is really just a slightly modified copy from a later avr-libc) into your hardware/arduino/cores/arduino directory also a partial solution?

[Bill52]

That would be asking PJRC to support non-PJRC products.  The Teensyduino installer is designed to never modify Arduino for non-PJRC boards.

At the moment, I'm working on a new Teensy product, so I don't have time for extra stuff lately.

I'm sorry, i must have misundertood something. From your 'long-email' below i gathered you are already supporting arduino.
I understood, your installer doesn't work for arduino boards because you respect the arduino-team's decision not to include your modifications.
Whatever, you viewpoint is taken.

Isn't simply copying the malloc.c I provided (which is really just a slightly modified copy from a later avr-libc) into your hardware/arduino/cores/arduino directory also a partial solution?

The "partial solution" was meant to say 'it is a solution until the arduino team provides a final fix'.

I don't know what is actually necessary to fix it.
I also don't get it, if it is so simple to fix it then why don't the Arduino people do that?
I  am not sarcastic here, i really don't see the issue: why isn't it fixed?

There seem to be an issue with dynamic mem.alloc. and String - which is dependent on malloc.
I can't remember the name - someone in the earlier posts said it is too hard to fix it.
And yes, i remember you did say to copy malloc.
Is this all to it?
One last thing. You said something about a compiler switch?
Is it a swith in the compiler to compile for arduino boards? Or for your boards?
Do i/we need to be concerned abouth this switch - whatever it may be.

Thanks for your time and effort anyway.

[pjrc]

I don't know what is actually necessary to fix it.

See my comment from July 27 on this page:

http://code.google.com/p/arduino/issues/detail?id=857

I also don't get it, if it is so simple to fix it then why don't the Arduino people do that?
I  am not sarcastic here, i really don't see the issue: why isn't it fixed?

That wasn't my decision.

As you can see on issue 468, I tried to contribute the fix in January 2011.

One last thing. You said something about a compiler switch?
Is it a swith in the compiler to compile for arduino boards? Or for your boards?
Do i/we need to be concerned abouth this switch - whatever it may be.

That patch is on issue 468.  It makes String more efficient.

I'm sorry, I busy with a big project, so I don't have time to write a detailed explanation.  I've already explained that patch many times before, so if you search enough, I'm sure you'll find one of those explanations.

[pjrc]

Also, if you do read my comment on issue 857, and grab the code from issue 468, you might also notice that so far absolutely nobody has done the optional step #4.

Next time you wonder why this bug still exists in Arduino for so very long, you might think about that lack of feedback?

[johncc]

Here's the bug report:

http://code.google.com/p/arduino/issues/detail?id=857

You may want to comment there about your belief that it should be urgently fixed. Maybe it'll move to "implemented" in version 1.0.2.

Perusing this thread and the bug report linked above I note that the bug tracking shows this has now finally been "fixed".

Comment 29 by project member c.mag...@bug.st, Dec 17 (2 days ago)
Done. Thank you.
https://github.com/arduino/Arduino/commit/d457332664730fde14146649169b1fdfe2209514
Please check the fix.

How will I know when this bug fix has made it into the official downloadable version 1.x.x?

Thanks,
John

(I hate C but love C++.  Mainly e.g. because of char* vs String)

[pjrc]

Unless it gets removed from the github repository (which would be very unusual), this should become part of 1.0.4.  When (or if) 1.0.4 will release is of course difficult to predict.

Version 1.5.2 will probably release soon, so in terms of official releases, it'll probably be available in the 1.5.x series before the 1.0.x series.

Of course, if you want it on 1.0.3 or earlier, all you have to do is copy the malloc.c file.  It's been used on Teensy since 0022, so it should work on all versions at least back to 0022.

[johncc]

Ok, thanks.

I remember last year when starting with Arduinos, finding that problem and eventually your malloc.c fix in the bug report.

A few months ago I discovered Teensy and have mostly been using them since...

Cheers,
John