Getting started with ARM code debugging

Hello everybody,

for the first time in many years of coding I finally found a situation that I really need to debug my code properly. Before today I always did debugging by adding dozens of additional Serial.print() in code to see where it crashes and always found a proble quite fast. I know that is quite stupid solution, but its quite always fast deployable.

Anyway right now I have a SAMD21G board (Arduino MKR Zero) with "quite" stable code that crashes in very unpredictable manner - once in 30 minutes, next time in several hours, next time in less than minute... In this case above "debugging" method does not work, since it seems that my code is simply stuck after my periodically run interrupt code is executed and it looks like it never returns back to loop.

So I have access to Atmel ICE programmer/debugger at work, as I read it should be possible to use it as debugger too.

I never debugged a code on MCU (only some code log time ago on x86 Linux), so I would like to know some basics on debugging on ARM that I did not find online.

1.) I assume that Atmel ICE can be used for debugging in combination with openocd and gdb?

2.) How can I compile & upload sketch from Arduino IDE with debugging symbols? I guess that without them any debugging would be useless...

3.) What kind of output can I get from debugger on such hardware? I can I just leave debugger connected for several hours and after crash just read in which function in crashed?

Any recommende additional resources for debugging basics/tutorials would be greatly appreciated!

Thank you

You need to spend some time reading some of the hundreds of available gdb tutorials. Your questions all deal with how to setup and use gdb, which really has nothing to do with Arduino.

Regards,
Ray L.

RayLivingston: You need to spend some time reading some of the hundreds of available gdb tutorials. Your questions all deal with how to setup and use gdb, which really has nothing to do with Arduino.

Regards, Ray L.

Well I know for some part your are definitely right.

But especially question 2 definitely IS RELATED to Arduino / Arduino IDE. Since I would like to ask someone who is familiar with debugging ARM MCUs to tell me to confirm / deny if I need to somehow alter build process (add debug symbols or something else) to get relevant debug output. As I know IDE does not have such options anywhere.

This question might sound silly but I have NO experience with debugging embedded hardware. I am NOT noob in general, but need to know if/how debugging on such hardware differs with standart "desktop" debugging....

isn't gdb used to debug code on a linux environment? I've used it in the past to identify a specific line in code that resulted in an exception. often tedious and a last resort

for an embedded application, logging some data would probably be helpful, in this case EEPROM. even prints to perhaps isolate where the problem lies. (and of course prints affect timing).

finding a way to exercise the fault more quickly is helpful

what about posting your code?

There is work in progress to add a debugger to the Arduino Pro IDE: https://github.com/arduino/arduino-pro-ide/issues/178 I suspect that the first architecture to be supported will be the SAMD boards, as that is Arduino's main product line right now.

Transfer your code to Atmel Studio for debugging.

I can debug Arduino sketches and libraries for SAMD in Arduino Elicpse plugin Sloeber.

gcjr: isn't gdb used to debug code on a linux environment?

gdb is available for many platforms, including Linux, Windows and OS/X, in both command-line and GUI versions

Regards, Ray L.

i guess debugging means different things to people

when programs were small, debuggers allowed you to step thru code and look at variables (registers). This becomes tedious when code is large or multiple tasks

many professional embedded programmers develop their code in simulation on a PC for example. i do this often enough to verify the logic of code targeted for arduinos. The code that reads or responds to I/O is tested on the target.

but the worst types of problems are when an exception occurs and the code dies, as described by the OP. ideally, a debugger is able to respond to an exception and capture the program counter to identify where exactly in the code the exception occurred.

i'd be surprised if a debugger running on a PC connected to an arduino processor via it's serial interface can do so. i know how difficult it is to debug such problems when the target runs linux and can capture core dumps.

Everyone on this thread is talking about on-chip debugging using a hardware debugger connected to either the SWD or JTAG interface of an ARM microcontroller.

gcjr: i'd be surprised if a debugger running on a PC connected to an arduino processor via it's serial interface can do so. i know how difficult it is to debug such problems when the target runs linux and can capture core dumps.

Without dedicated on-chip debug hardware, there is little you CAN do other than insert print statements, toggle outputs, etc. You cannot, with rare exceptions that do not apply to most AVR-Arduinos, do single-step, breakpoints, etc.

That said, I've caught and fixed many bugs that would occur as rarely as once in several days with nothing but print statements. But it can take weeks to pin-point the source that way.

Regards, Ray L.

RayLivingston: That said, I've caught and fixed many bugs that would occur as rarely as once in several days with nothing but print statements. But it can take weeks to pin-point the source that way.

the "art" of debugging

gcjr: the "art" of debugging

Yeah.... I once spent three months tracking down a really insidious bug that was caused by both four tiny logic errors in four different parts of the chip itself, as well as problems in the driver software. The failure required a sequence of 5-6 events to occur, in sequence with each of those events having a one-clock window were it could propagate the fault. When that sequence occurred, which it did with surprising frequency given the complexity of the sequence, data was corrupted or lost with no error being flagged. THAT was a looooooong three months! A few simple changes to the logic, and a quick re-spin of the chip (fortunately a metal mask only change), and it worked perfectly.

Regards, Ray L.

Hello pert & juraj, glad to see you again ! :)

This thread seems to be more alive than I have expected. In recent year or so I am really surprised how Arduino community grew from "bunch of garage DIYers" to state that we can discuss really advanced issues and serious programming stuff.

Arduino IDE Pro looks really awesome. I am missing option to set fonts size (really tiny on my huge monitor), but from functional point there are significant improvements. Thanks for that, I completely missed this information !

Anyway, back to thread :)

I have been successfull in trial & error experiments and connected gdb to ICE debugger. I have noticed two different crashes. Because I am coding on Linux machine, I have noticed that sometimes my board is giving various errors in standart dmesg system log, like:

[322287.498420] usb 1-1: device descriptor read/64, error -71
[322287.734508] usb 1-1: device descriptor read/64, error -71
[322287.842365] usb usb1-port1: attempt power cycle
or
[312780.566266] cdc_acm 1-1:1.0: ttyACM0: USB ACM device
[313089.812601] cdc_acm 1-1:1.0: failed to set dtr/rts
or similar...

These messages can tell me that code on MCU has crashed completely. I verified that. My guess is that USB PHY needs some kind of interraction with core so if code crashes and core is stuck, so is USB interface. Thats my guess that seems to cover my problems. Not sure if its 100% correct. If my host system log is clear of errors related to USB, then (again) my guess is that code is actually running in some unexpected cycle or state.

In my latest experiments I discovered one tiny coding error that might cause out of defined array memory write. Code is running for some time now, we will see if it runs over an 1 hour... I really hope this is what is causing complete death of MCU.

Anyway, I will probably write some blog post (?) about my initial experience with gdb od SAMD21. It really is simple to deploy. But what you might not read (and imagine when reading and trying that) is that all tutorials using "load" function in gdb to load program into MCU. I actually never had success. I always soft-broke my board and had to reflash bootloader :D I guess that there is specific load address offset or some mess with fuses. I ended with basically flashed code from Arduino IDE in standart way and when done connected to ICE via gdb with loaded debug symbols separatedly. That worked fine.

So I hope I am done (will let code run over night).

I have one question. From what I understand, am I right that there is no way to detect that my code got stuck in loop other than adding prints? I am processing high speed serial data and I have a lot of while() loops waiting for exact packets of bytes. These functions are often nested and very time sensitive. Are there any other options than getting a strong coffee and enjoy day with staring and brainstorming on the code? My other idea was adding an interrupt on button push, but it would be only usable if you could get information where program was before interrupt...

1.) I assume that Atmel ICE can be used for debugging in combination with openocd and gdb?

I think so. https://www.omzlo.com/articles/programming-the-samd21-using-atmel-ice-with-openocd-(updated)

2.) How can I compile & upload sketch from Arduino IDE with debugging symbols? I guess that without them any debugging would be useless...

Most Arduino SAMD platforms already include the "-g" switch to enable creation of debug symbols (or have it in an option in the "board" menu.) This puts debugging information into the .elf file, which is what you'd normally load into gdb. The code loaded into the actual board (.bin or .hex) doesn't have any debugging info, but it's essentially just a stripped version of the .elf, so the debugging software can match them up.

3.) What kind of output can I get from debugger on such hardware? I can I just leave debugger connected for several hours and after crash just read in which function in crashed?

Yes, but... The hard part is detecting this mysterious "crashed" state. See my other recent post on the same subject

am I right that there is no way to detect that my code got stuck in loop other than adding prints? I am processing high speed serial data and I have a lot of while() loops waiting for exact packets of bytes.

You can reset a timer in some basic place (beginning of loop(), for instance) for a time period longer than you expect any (or all of) function to take. Your debugger can put a breakpoint in the ISR for the timer, and if some code takes much longer than expected, you'll hit that breakpoint. This is the sort of thing that the Watchdog timer is designed to do, except that it's frequently set up to completely reset the system when the timer expires (and it might be easier to use some other timer.)

westfw: Yes, but... The hard part is detecting this mysterious "crashed" state. See my other recent post on the same subject

Thanks, really interesting and quite importatnt to me to know that ARM behaves differently than AVR I used a lot of years.

westfw: You can reset a timer in some basic place (beginning of loop(), for instance) for a time period longer than you expect any (or all of) function to take. Your debugger can put a breakpoint in the ISR for the timer, and if some code takes much longer than expected, you'll hit that breakpoint. This is the sort of thing that the Watchdog timer is designed to do, except that it's frequently set up to completely reset the system when the timer expires (and it might be easier to use some other timer.)

Good idea, I implemented that and fixed some issues. Still have something to be fixed. I assume that if my code executes the interrupt function code (that is redrawing LCD) it actually still runs even the main loop looks dead because its not displaying any serial debug output.

So I have a question - is ARM handling local variables and arrays declared in loop differently? Could it be affected by interrupt?

Because now I am sure that problem is that interrupt may (in very rare case after several hours) be called in the beginning (my guess) of my Serial parsing function in main loop and somehow looping that.

I cannot post code because this function cannot fit into post size limit...