ESP32, in-circuit debugging

Software IDE 2.x
1

I read that EsspressIF migrated away from their extended version of gdb and now support 'standard' gdb and since the Arduino IDE v2.x.x has gdb then what will it take to get in-circuit debugging happening with the ESP32 boards?

I'm asking because I have a project which used a Nano with 2 add-on boards(Bluetooth and RTC) and I was running out of program space which required me to limit the usable term of the project. And ESP32 seems a good target to upgrade to and since I've seen how well the debugging works with the Zero I'd like to ditch my debug macros and do in-circuit debugging on the ESP32.

if another gdb debugger is needed, there's xtensa-esp32-elf-gdb which could be launched and is what's used by Eclipse.

While searching for a gdb/Arduino IDE 2 solution I ran across lots of mentions of gdbstub.h and a serial interface debugging capability for AVR 328 Arduino variants and wonder it that's an option here as well. Having an in-circuit debugger on the Arduino platform is a big deal so I hope there are ways to get more than the current supported boards working.

2

Hi @Dougl. Even though currently it is only used with the boards of the official "Arduino SAMD Boards (32-bits ARM Cortex-M0+)" platform as far as I know, the Arduino IDE 2.x integrated sketch debugger is set up to be usable with the boards of any platform, official or 3rd party.

There are a couple of requirements:

  • The board you are using is compatible with the free open source "Cortex-Debug" extension that provides the integrated sketch debugger in Arduino IDE 2.x.
  • The platform of the board has been configured for use with the debugger.

Let's look at the first condition: as the name of the extension indicates, Cortex-Debug targets microcontrollers with an ARM Cortex-M IP core. ESP32 uses a Tensilica core (or RISC-V in the case of the ESP32-C3) instead of Cortex-M, so that is not good news for the prospects of compatibility with the debugger. However, even though the author of the debugger targeted Cortex-M, it is possible other microcontrollers might be compatible as well, as discussed here:

I think it would be worth it for someone to do some experimentation to see whether it is possible. Note that Cortex-Debug is a VS Code extension, so you are welcome to install and use it in VS Code if it is more convenient to do the initial experimentation in that environment.

Now for the second condition: the ESP32 platform developers have not added the necessary configuration properties to the platform files. The only Arduino boards platform I know of that has been configured for use with the debugger is the official "Arduino SAMD Boards (32-bits ARM Cortex-M0+)" platform, even though many other 3rd party platforms are for Cortex-M based microcontrollers that are definitely compatible with the debugger. At least for that platform, it only took adding a small number of lines to the boards.txt and platform.txt files to add debugger support. So that platform can serve as a model for anyone wanting to configure other platforms.

3

Thanks @ptillisch for that information. I'll try a few things to see how far I can get. Besides finding the gdbstub.h solution for avr328 devices I see that STM32 devices can be made to work with openOCD and the ST-Link dongle by manipulating platform.txt and boards.txt files as described here:

https://www.stm32duino.com/viewtopic.php?f=18&p=10740

I have an STM32 bluepill and will try this option to help me get my head around the procedures/processes involved to maybe help with the ESP32 platform. I found the 2 boards I'm currently working with don't have JTAG or SWD access so will look at some of the other ESP32 boards to find these to test with.

Besides the ST-Link interface I also have a BlackMagic Probe device which houses its own gdb server so that might be an option instead of openOCD/ST-Link and since BMP can be uploaded to a $10 ST-Link device it could open up more debugging options to more people than $75+ for official BMP or other debuggers.

Planning:

  1. get OpenOCD/ST-Link debugging working with IDE v2.0.3 and STM32
  2. get BlackMagic Probe debugging working with IDE v2.0.3 and STM32
    update:
  3. get OpenOCD/esp-prog debugger working with IDE v2.0.3 and ESP32
4

I think getting debugging working with the STM32 in Arduino IDE 2.x will likely be straightforward since these microcontrollers use an ARM Cortex-M core.

I'm glad to see there is some movement towards adding support to the STM32duino platform. I have been a bit surprised that I haven't seen any 3rd party boards platforms take advantage of the new debugger yet.

I have also been quite interested in seeing whether it might be possible to use the Black Magic Probe with Arduino IDE 2.x. It seems like a great open source project in a space where closed source hardware, firmware, and even software is still dominant.

If you think of it, please post an update here with your findings.

1 Like
5

I'm getting close. I did very similar to the STM32 bits to add scripts and enable debugging in the UI. I also had to add the esp-if tools to get some openocd config files AND I had to update platform.txt to use openocd 0.11 from the esp-if tools unlike the stm32 debugging which used to Arduino openocd v0.10 without issue.

Openocd starts but halts out and it looks like it's not getting flash maps. When I run the same openocd command(copied from IDE console) on the commandline( with idf env vars set ) it runs fine.

I retested the stm32 blink project to see if the debugger/openocd v0.11 still worked and it worked as before with step into, over, continue etc working fine.

How does one add a menu to the Tools section? The STM32 and Zero had options to turn on compiler flags for debugging and the ESP32 does not have those.

6

You do it by editing the definition of the board you are using it its platform's boards.txt file.

You can learn about this "custom board options" system here:

https://arduino.github.io/arduino-cli/latest/platform-specification/#custom-board-options

Something you should be aware of is that there is a bug in Arduino IDE 2.x that causes newly added custom board options to not show up in the menu:

The workaround for that bug is provided here:

https://github.com/arduino/arduino-ide/issues/1030#issuecomment-1152005617

If you are talking about changing the optimization settings, there is a dedicated menu item for this purpose at Sketch > Optimize For Debugging. This system is documented here:

https://arduino.github.io/arduino-cli/latest/platform-specification/#sketch-debugging-configuration

Please let me know if you have any questions or problems while setting up custom board options or the debug optimization properties.

7

Thanks, I'll check those out and see what I can get going.

I'm also going to post about my adventures on the esp32.com forum incase there's someone there with understanding of what might be happening.

Deleting the .config/arduino-ide/ folder does indeed reset the cache so the menu changes take effect.

I also found that the "Sketch->Optimize for Debugging" doesn't add debug symbols or compiler options and instead it optimizes the building so it's not building everything and just builds the local project files as needed. Far quicker turn-around for code changes and retesting.

A mear copy and paste of the optimization menu section from the stm32 GenF1 board does put the menus in place in the esp32 but it's not triggering the compiler flag changes so there's something else going on to get flags passed into the compiler section.

8

Let's take a look at one option in the "Optimize" custom board option menu (menu ID opt):

https://github.com/stm32duino/Arduino_Core_STM32/blob/2.3.0/boards.txt#L7300

GenF1.menu.opt.ogstd=Debug (-Og)

This line adds a "Debug (-Og)" option to the Tools > Optimize menu in Arduino IDE.

https://github.com/stm32duino/Arduino_Core_STM32/blob/2.3.0/boards.txt#L7301

GenF1.menu.opt.ogstd.build.flags.optimize=-Og

This line causes a build property named build.flags.optimize to be defined and set to the value -Og when the user selects Tools > Optimize > Debug (-Og) from the Arduino IDE menus.

This build.flags.optimize is only an arbitrary property. Defining it doesn't do anything on its own. The property must be used somewhere. You can see one of the places it is used here:

https://github.com/stm32duino/Arduino_Core_STM32/blob/2.3.0/platform.txt#L37

compiler.cpp.flags={compiler.extra_flags} -c {build.flags.optimize} {build.flags.debug} {compiler.warning_flags} -std={compiler.cpp.std} -ffunction-sections -fdata-sections -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -fno-use-cxa-atexit -MMD {compiler.stm.extra_include}

(note the {build.flags.optimize} property reference in that line)

This property reference is being used as part of the definition of yet another build property: compiler.cpp.flags. Once again, defining that property doesn't have any effect on its own if it is not used somewhere. You can see it being used here:

https://github.com/stm32duino/Arduino_Core_STM32/blob/2.3.0/platform.txt#L129

recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpp.flags} {build.info.flags} {compiler.cpp.extra_flags} {build.st_extra_flags} {build.extra_flags} {compiler.arm.cmsis.c.flags} {includes} "{source_file}" -o "{object_file}"

(note the {compiler.cpp.flags} property reference in that line)

The previous build properties were only arbitrary properties, which the platform developer can define as many as they like with any name, and reference them as they like. There are also some properties that have special significance in the Arduino boards platform framework. recipe.cpp.o.pattern is one of them. This defines the "pattern" (i.e., template) used to generate one of the commands Arduino IDE runs during the compilation:

https://arduino.github.io/arduino-cli/latest/platform-specification/#recipes-to-compile-source-code

So the combination of setting the property in boards.txt and then referencing it (indirectly) in the recipe.cpp.o.pattern causes the -Og flag to be added to the compilation command.

This is likely because you aren't referencing the {build.flags.optimize} property in the appropriate places in your platform.txt file.

1 Like
9

Thanks for that thoroughly detailed description and it's exactly what I had found last night . I tried doing it with the esp32 kit and did not see the compiler option on the IDE console as I do when using the stm32 code. I went after the {build.flags.optimize} definitions. I gained an appreciation for the settup effort the stm32 devs put forth vs the esp32 in as far as readability of the boards.txt and platform.txt setup files go.

Update: Thanks for putting me back on the track! I looked what I was doing in platform.txt and noticed there was a hardcoded -Os later in the compiler definitions. I found it by not scrolling through the output this time but by copy/pasting the output into the kate editor and searching for "-O".
Sure enough, looking at what I pasted below from the platform.txt section you can see the -Os later in the line.

stm32 menu:

GenF1.menu.opt.osstd=Smallest (-Os default)
GenF1.menu.opt.oslto=Smallest (-Os) with LTO
GenF1.menu.opt.oslto.build.flags.optimize=-Os -flto
GenF1.menu.opt.o1std=Fast (-O1)
GenF1.menu.opt.o1std.build.flags.optimize=-O1
GenF1.menu.opt.o1lto=Fast (-O1) with LTO
GenF1.menu.opt.o1lto.build.flags.optimize=-O1 -flto
GenF1.menu.opt.o2std=Faster (-O2)
GenF1.menu.opt.o2std.build.flags.optimize=-O2
GenF1.menu.opt.o2lto=Faster (-O2) with LTO
GenF1.menu.opt.o2lto.build.flags.optimize=-O2 -flto
GenF1.menu.opt.o3std=Fastest (-O3)
GenF1.menu.opt.o3std.build.flags.optimize=-O3
GenF1.menu.opt.o3lto=Fastest (-O3) with LTO
GenF1.menu.opt.o3lto.build.flags.optimize=-O3 -flto
GenF1.menu.opt.ogstd=Debug (-Og)
GenF1.menu.opt.ogstd.build.flags.optimize=-Og
GenF1.menu.opt.o0std=No Optimization (-O0)
GenF1.menu.opt.o0std.build.flags.optimize=-O0

esp32 menu:

esp32.menu.opt.osstd=Smallest (-Os default)
esp32.menu.opt.oslto=Smallest (-Os) with LTO
esp32.menu.opt.oslto.build.flags.optimize=-Os -flto
esp32.menu.opt.o1std=Fast (-O1)
esp32.menu.opt.o1std.build.flags.optimize=-O1
esp32.menu.opt.o1lto=Fast (-O1) with LTO
esp32.menu.opt.o1lto.build.flags.optimize=-O1 -flto
esp32.menu.opt.o2std=Faster (-O2)
esp32.menu.opt.o2std.build.flags.optimize=-O2
esp32.menu.opt.o2lto=Faster (-O2) with LTO
esp32.menu.opt.o2lto.build.flags.optimize=-O2 -flto
esp32.menu.opt.o3std=Fastest (-O3)
esp32.menu.opt.o3std.build.flags.optimize=-O3
esp32.menu.opt.o3lto=Fastest (-O3) with LTO
esp32.menu.opt.o3lto.build.flags.optimize=-O3 -flto
esp32.menu.opt.ogstd=Debug (-Og)

esp32 platform.txt:

compiler.c.flags.esp32=-mlongcalls -c {build.flags.optimize} -Wno-frame-address -ffunction-sections -fdata-sections -Wno-error=unused-function -Wno-e
rror=unused-variable -Wno-error=deprecated-declarations -Wno-unused-parameter -Wno-sign-compare -ggdb -Os -freorder-blocks -Wwrite-strings -fstack-pr
otector -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu99 -Wno-old-style-declara
tion -MMD -c
compiler.cpp.flags.esp32=-mlongcalls -c {build.flags.optimize} -Wno-frame-address -ffunction-sections -fdata-sections -Wno-error=unused-function -Wno
-error=unused-variable -Wno-error=deprecated-declarations -Wno-unused-parameter -Wno-sign-compare -ggdb -Os -freorder-blocks -Wwrite-strings -fstack-
protector -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu++11 -fexceptions -fno-
rtti -MMD -c

UPDATE2: I've now also been able to pass both optimization flags and debug flags and I know why I did not see the double optimization flags on the IDE console, it was too long and it was being visually concatenated. Here I've added both the optimization and debug flags:

compiler.c.flags.esp32=-mlongcalls -Wno-frame-address -ffunction-sections -fdata-sections -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wno-unused-parameter -Wno-sign-compare -ggdb {build.flags.optimize} {build.flags.debug} -freorder-blocks -Wwrite-strings -fstack-protector -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu99 -Wno-old-style-declaration -MMD -c

10

Now I have to get to the bottom of why the openocd is erroring out:

Waiting for gdb server to start...[2022-12-19T16:49:10.008Z] SERVER CONSOLE DEBUG: onBackendConnect: gdb-server session connected. You can switch to "DEBUG CONSOLE" to see GDB interactions.
/home/maker/.espressif/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin/openocd -c "gdb_port 50000" -c "tcl_port 50001" -c "telnet_port 50002" -s /home/maker/Arduino/Blink-esp32Dev -f /home/maker/Projects/Arduino/arduino-ide_nightly-20221217_Linux_64bit/resources/app/plugins/cortex-debug/extension/support/openocd-helpers.tcl -f /home/maker/.arduino15/packages/esp32/hardware/esp32/2.0.5/variants/esp32/openocd_scripts/esp-wroom-32.cfg
Open On-Chip Debugger v0.11.0-esp32-20221026 (2022-10-26-14:47)
Licensed under GNU GPL v2
For bug reports, read
OpenOCD: Bug Reporting
CDRTOSConfigure
Warn : Transport "jtag" was already selected
Info : Listening on port 50001 for tcl connections
Info : Listening on port 50002 for telnet connections
Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi tdo_sample_edge falling"
Info : clock speed 20000 kHz
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: auto0.tap tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Warn : AUTO auto0.tap - use "jtag newtap auto0 tap -irlen 5 -expected-id 0x120034e5"
Info : starting gdb server for esp32.cpu0 on 50000
Info : Listening on port 50000 for gdb connections
Info : accepting 'gdb' connection on tcp/50000
Info : [esp32.cpu0] Debug controller was reset.
Info : [esp32.cpu0] Core was reset.
Info : [esp32.cpu0] Target halted, PC=0x400832F0, debug_reason=00000001
Info : [esp32.cpu0] Reset cause (3) - (Software core reset)
Warn : No symbols for FreeRTOS!
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Error: Failed to get flash maps (4294967290)!
Warn : Failed to get flash mappings (-4)!
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Info : Auto-detected flash bank 'esp32.cpu0.flash' size 4096 KB
Info : Using flash bank 'esp32.cpu0.flash' size 4096 KB
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Error: Failed to get flash maps (4294967290)!
Warn : Failed to get flash mappings (-4)!
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Info : Using flash bank 'esp32.cpu0.irom' size 0 KB
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Error: Failed to get flash maps (4294967290)!
Warn : Failed to get flash mappings (-4)!
Info : [esp32.cpu0] Target halted, PC=0x40092612, debug_reason=00000001
Info : Using flash bank 'esp32.cpu0.drom' size 0 KB
Info : dropped 'gdb' connection
shutdown command invoked
[2022-12-19T16:49:11.002Z] SERVER CONSOLE DEBUG: onBackendConnect: gdb-server session closed
GDB server session ended. This terminal will be reused, waiting for next session to start...

11

@ptillisch if you have not seen it yet, the 2.0.6 update of the ESP Core includes working JTAG debugging capabilities! I tried it with an FT232H adapter, selecting the new FTDI debugger option and it worked. I don't see any menu to set debug code generation options so I will have to see how that is handled but a real debugger is now possible with ESP32!

12

Thanks @Dougl!

I did notice this when I updated to 2.0.6 because the platform now automatically adds a debug_custom.json configuration file to the sketch when you compile for an ESP32 board. It was quite a pleasant surprise.

I'm really happy to see more boards platforms taking advantage of this capability of Arduino IDE 2.x. Seeing it in the ESP32 platform made me think of Earle F. Philhower, III's RP2040 platform, which seems to be somewhat modeled after the ESP32 platform. Sure enough, I discovered debugger support was added to that platform as well:

I haven't tried either yet, but I am looking forward to it. I saw mention of the "ESP-Prog" debug probe being based on an FTDI chip, and was thinking I might be able to DIY one, but didn't find any information about that after a quick search. I did see that even the official ESP-Prog hardware is very reasonably priced and well documented so that is a good option as well and I'll definitely order one. But I already have FT232R modules on hand so I'll have to look into that more.

13

Hi @ptillisch, I have on order an ESP-Prog board but while waiting I picked up an FT232H and that has worked well since the FT2232H is what's doing the work in the ESP-Prog board. I don't know if the FT232R is the same but if it is, then it should work.

I only connect up gnd and the 4 JTAG lines so that I can still get serial out and power from the USB interface since the 3.3V on the RT232H might not be enough for esp32 wireless hardware.
FT232H adapter

I'm not sure what this means for debugging or compiler options but I will look into it. I can't imagine the default is to build with debug symbols for all builds. I looked for any compiler debugging directives in an IDE menu, as I had built from the STM32 example, and there were none.

I'll be passing on the word about all the JTAG debugging capabilities in the v2 IDE. Gotta get to the bottom of the SPIFFS( or better LittleFS ) uploading issue since the ESPs have relied on a filesystem for HTML services.

Wow, there are a bunch of new files in the sketch folder besides the developer sketch now:
debug.cfg debug_custom.json esp32.svd

14

I haven't looked closely to understand why they generate the debug_custom.json file.

When using the Cortex-Debug extension in VS Code, you would add a file named launch.json to the project to configure the debugger:

https://github.com/Marus/cortex-debug/wiki#vscode-settings-for-cortex-debug

Arduino users expect an easier workflow so Arduino IDE automatically configures the debugger extension based on the configuration in the platform of the selected board.

You can see the auto-generated launch.json that is used by the debugger by doing this:

  1. Click the "Start Debugging" button on the button bar.
  2. Wait for the "DEBUG" view to open in the side bar.
  3. Click the 'Open "launch.json"' icon (:gear:).

A launch.json tab will open in the IDE editor. Note this is read-only, so you can view it, but not modify it.

But in some cases it might be necessary or useful to allow the user to make some customizations. So the Arduino developers provide the option of adding a file named debug_custom.json to the sketch. The contents of that file are merged into the configurations[0] key of the launch.json file.

Historically, the most common use of debug_custom.json is for people who want to use a J-Link debug probe. Segger's tools are proprietary and so can't be distributed with the open source Arduino boards platform. The user must install these separately. This means the path to the tool can't be determined from the platform configuration as usual. It is then necessary to configure the debugger to know the path to the tool:

https://docs.arduino.cc/tutorials/mkr-wifi-1010/mkr-jlink-setup#installing-the-j-link-gdb-server-software

This is definitely a problem. The lack of support for such plugins is tracked by the Arduino IDE developers here:

Since Arduino has not shown any progress on that, it looks like the ESP platform developers may be plowing on to try to implement a solution on their own:

https://github.com/arduino/arduino-ide/issues/58#issuecomment-1354775556

We are now in an odd situation where only Arduino IDE 1.x provides the plugin capability that is much used by the users of the Espressif chip boards, but only Arduino IDE 2.x provides the debugger capability that is likely to become an important feature for the same community.

1 Like
16

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.