Sharing Debugging Techniques

[LarryD]

Here is one hint you might not have seen:

• Every now and then we need to see if a certain section of code is running.

• During debugging, we often use Serial.print() to display a message or a variable's value.

• I always add a "Copy Tab" to my project sketches.
  I have commonly used debugging lines of code in this Tab so I don't have to retype things.

• One such example is code to print how many times we reach a code section.

//Prints the number of times we have been at this location. 
static unsigned int counter = 1;                        // <---<<<<  D e b u g
Serial.print("Counter = ");                             // <---<<<<  D e b u g
Serial.println(counter++);                              // <---<<<<  D e b u g

//Prints the time it takes to return to this position.  // <---<<<<  D e b u g
static unsigned long startTime;                         // <---<<<<  D e b u g
Serial.println(micros() - startTime);                   // <---<<<<  D e b u g
startTime = micros();                                   // <---<<<<  D e b u g

//Prints the time it takes to run this section of code. // <---<<<<  D e b u g
static unsigned long startTime;                         // <---<<<<  D e b u g
startTime = micros();                                   // <---<<<<  D e b u g
//Section of code                                       // <---<<<<  D e b u g
Serial.println(micros() - startTime);                   // <---<<<<  D e b u g

---

• I add <---<<<< D e b u g to the end of debug lines so they can be easily identified.

• Always remember, when we add debugging code to a sketch, it may effect that sketch, ex: might change timing.

How about you software people share some hints.

[sumguy]

Depending on what I am doing I might include "debug switches" that I can turn on or off.

#define Debug_1 0
#define Debug_2 1
#define Debug_3 0
#define Debug_4 0

int cnt=0;

void setup() {
  // put your setup code here, to run once:
 Serial.begin(9600);
}

void loop() {
  // put your main code here, to run repeatedly:
  cnt+=1;
  delay(300);
  if(Debug_2){Serial.prinln(cnt);}
}

[ptillisch]

The system I use:

Add this code near the top of your sketch:

#define DEBUG true  // Set to true for debug output, false for no debug output.
#define DEBUG_SERIAL \
  if (DEBUG) Serial

Then use code like this to initialize the serial communication:

DEBUG_SERIAL.begin(9600);

If you want your program to wait for Serial Monitor to be opened before running when using native USB boards (e.g., Leonardo), add this line:

#if DEBUG == true
  while (!Serial) {
    ;  // wait for serial port to connect. Needed for native USB port only
  }
#endif  // DEBUG == true

and code like this for debug output:

DEBUG_SERIAL.println("Some debug output");

When the DEBUG macro is set to false, the compiler will optimize the calls using DEBUG_SERIAL out of the code because it knows they will never run. This means the debug output code won't use up any memory and won't slow down the execution of the program. This means you can leave the debug code in place in the production code.

In the rare case where your debug system needs to read serial input, you would use similar code:

#if DEBUG == true
  if (Serial.available()) {
    x = Serial.read();
  }
#endif  // DEBUG == true

---

This system can easily be extended to allow multiple levels of debug output, still with no overhead when it's disabled:

#define DEBUG_ERROR true
#define DEBUG_ERROR_SERIAL \
  if (DEBUG_ERROR) Serial

#define DEBUG_WARNING true
#define DEBUG_WARNING_SERIAL \
  if (DEBUG_WARNING) Serial

#define DEBUG_INFORMATION true
#define DEBUG_INFORMATION_SERIAL \
  if (DEBUG_INFORMATION) Serial

void setup() {
  Serial.begin(9600);
  while (!Serial) {
    ;  // wait for serial port to connect. Needed for native USB port only
  }
  DEBUG_ERROR_SERIAL.println("This is an error message");
  DEBUG_WARNING_SERIAL.println("This is a warning message");
  DEBUG_INFORMATION_SERIAL.print("The state of pin 5 is ");
  DEBUG_INFORMATION_SERIAL.println(digitalRead(5) ? "HIGH" : "LOW");
  Serial.println("This is standard program output");
}

void loop() {}

As with @sumguy's suggestion, my Boolean debug macros can be used to switch on and off other parts of your code too.

[embeddedkiddie]

Although it's purely a hobby, I've recently been writing somewhat complex programs on ESP32, so I've been keeping the following in mind:

Start small, test small

Especially when combining multiple algorithms or libraries, instead of writing the combined code and then testing it, test each one in a small dedicated sketch and then combine them bit by bit.

It may seem tedious, but I've found that it makes debugging easier in the end.

Better safe than sorry

From a design perspective, I use assert() in advance to check what should and shouldn't happen. When testing various use cases with some hardware, I often find unexpected bugs.

It's useful because it displays the source file name and line number.

#define	DEBUG true
#if DEBUG
#define	DBG_ASSERT(x)	assert(x)
#else
#define	DBG_ASSERT(x)
#endif

Print something more complex

Sometimes I want to do something more complex than simply outputting values ​​to the serial console using print functions.

To do this, I use the following macros:

#define	DEBUG true
#if DEBUG
#define	DBG_EXEC(x)	x
#else
#define DBG_EXEC(x)
#endif

DBG_EXEC({
  printf("ID: %d\n", controller.id);
  for (auto &i : controller.items) {
    printf(" name: %s, hash: 0x%x\n", i.name.c_str(), i.hash);
  }
});

In either case, if it's a hobby program for personal use only, there's no problem with leaving the debugging code behind, but since I often publish my work on GitHub, I tend to make sure to remove the debugging code.

Bonus: Analyze unexpected exception

The EspExceptionDecoder and esp-exception-decoder are great tools to trace the cause of unexpected exception such as a "Guru meditation error", but unfortunately they don't work with Arduino IDE 2.x.

So I use the ESP Stack Trace Decoder hosted on the GitHub page.

All I have to do is to drop the .elf file from the sketch's cache directory and double-click the Backtrace: line in the Output console to copy and paste.

[gilhad]

This times I often look into disassembled code, to see, how some parts are exactly compiled and if they do exactly what I want.

Also I often run the code in emulator, set breakpoints to some interesting places, then let it run and when it hit the breakpoint I step it by instructions to see, what happens.

Disassembler for ATmega2560 (Arduino Mega R3)

I am writing larger and complicated project for ATmega2560 and I wanted to see, what is the resulting code. I was using avr-objdump for long time, but still wanted something better. Yesterday I found this project that had good outputs, but lacked support for ATmega2560, so I cloned it, added this support and improve output (variable names and offsets mainly) Here it is on github https://github.com/githubgilhad/AVRemu It show names of variables (like RX_Count), HW registers (like UBRR0H and U…

I placed some code to init3 to fill all memory with 0xA5 (low memory) and ‘H’ (high/external memory) and now I see, what was used.

#include <avr/io.h>
;	.init0 first part on boot, no stack
;	.init1 
;	.init2 SP defined
;	.init3 <======================== HERE we are
;	.init4 fill .data from FLASH
;	.init5 fill .bss with 0
;	.init6
;	.init7
;	.init8
;	.init9 call main
.section .init3
.global fill_ram_pattern
fill_ram_pattern:
	ldi r30, lo8(__heap_start)
	ldi r31, hi8(__heap_start)
	
	ldi r26, lo8(__stack)
	ldi r27, hi8(__stack)
	
	ldi r24, 0xA5
	
1:
	cp  r30, r26
	cpc r31, r27
	brsh 2f
	
	st  Z+, r24
	rjmp 1b
2:

;	MCRA = (1<<SRE);     // Enable External Memory Interface
;	XMCRB = 0x00;

	sbi _SFR_IO_ADDR(DDRG),  3
	sbi _SFR_IO_ADDR(PORTG), 3		; XAA16 on (select high half of 128kB RAM chip)
	ldi r24, 1<<SRE
	sts XMCRA, r24		; set bit SRE
	clr r24
	sts XMCRB, r24		; clear XMCRB

fill_high_ram_pattern:
	ldi r30, lo8(0x2200)
	ldi r31, hi8(0x2200)
	
	clr r26
	
	ldi r24, 'H'
	
1:
	cp  r30, r26
	cpc r31, r26
	breq 2f		; overflow from 64kB to 0x0000
	
	st  Z+, r24
	rjmp 1b
2:

also I fill my stacks and buffers with easily recognizable pattern like this:

	for (uint8_t i=0; i<DST_SIZE;++i)	TCB_test.DataStack[i] 	= u32_to_p24(0x444444); // 'DDD'
	for (uint8_t i=0; i<RST_SIZE;++i)	TCB_test.ReturnStack[i]	= u32_to_p24(0x525252); // 'RRR'
	for (uint8_t i=0; i<LST_SIZE;++i)	TCB_test.LStack[i]	= u32_to_p24(0x4C4C4C); // 'LLL'
	for (uint8_t i=0; i<TIB_SIZE;++i)	TCB_test.TIB[i]	= 'T';
	for (uint8_t i=0; i<AIB_SIZE;++i)	TCB_test.AIB[i]	= 'A';
	for (uint8_t i=0; i<ORDER_SIZE;++i)	TCB_test.WL_ORDER[i]	= u32_to_p24(0x4f4f4f); // 'OOO'
	for (uint16_t i=0; i<HERE_SIZE;++i)	HERE1[i]	= 'H';

and I can easy see, how much of which stack was ever used (ops, ReturnStack should starts about 1 position higher and keeps 1 pointer to FLASH 0x008810, and ORDER keeps 3 pointers (24bit each) 0x800304, 0x800301 and 0x8002fe which both is expected)

0700: 52 52 52 52  52 10 88 00  52 52 52 4c  4c 4c 4c 4c     RRRRR...RRRLLLLL
0710: 4c 4c 4c 4c  4c 4c 4c 4c  4c 4c 4c 4c  4c 4c 4c 4c     LLLLLLLLLLLLLLLL

07e0: 41 41 41 41  41 41 41 41  41 41 41 41  41 41 41 41     AAAAAAAAAAAAAAAA
07f0: 41 41 41 41  41 41 41 41  41 41 41 41  41 41 41 41     AAAAAAAAAAAAAAAA
0800: 41 41 41 41  41 04 03 80  01 03 80 fe  02 80 4f 4f     AAAAA.........OO
0810: 4f 4f 4f 4f  4f 4f 4f 4f  4f 4f 4f 4f  4f 4f 4f 4f     OOOOOOOOOOOOOOOO
0820: 4f 4f 4f 4f  4f 4f 4f 4f  4f 4f 4f 4f  4f 4f 4f 4f     OOOOOOOOOOOOOOOO

[Dave_Lowther]

If I’m using a board that supports a file system, e.g. ESP32 devboard, I always include a software error log file. Most of my ESP32 projects have a web UI with an option to view the error log.

Example:

That’s just a normal (no errors) boot sequence. I also have error logging for when functions that should return success actually fail and E.G. for default: in case statements where the default case should never happen. I find the log file information helps me to investigate rare bugs that happen in always on devices.

[gcjr]

i know there's NOT one way to debug a problem.

• Sometimes it's just getting the code to work,
• at other times it's a more insidious problem that requires seeing prints when the corner case arises,
• need logs to analysis the states of variables off-line

too many debug prints can make things more difficult to debug. Need to focus on the section of code with the problem

sometimes prints need to be temporary

• added but easily commented out
• conditionally executed using a dbg flag
• conditionally executed using debug bits e.g. if (dbg & 4)
• deleted once the code is working properly

sometimes the time to print takes too long and affects the code causing problems., trace buffer is needed, dumped after some event occurs and analyzed off-line. Often compact binary codes instead of text. Sometimes that trace buffer requires timestamps as well

Serial.println(__func__); at the top of each function shows the flow through each function

numerous Serial.println(" funcname - #") within a function with # incremented for each can help identify where code crashes within a function

instead of numerous Serial.print()s use sprintf(s, ...); Serial.println(s); to minimize the # of debug lines (esp32supports printf())

[LarryD]

• Add these lines to to a spot in your sketch to see if it gets there:

  //Print the line we are currently at:   // <---<<<<  D e b u g
  Serial.print("Line Executed = ");       // <---<<<<  D e b u g
  Serial.println(__LINE__);               // <---<<<<  D e b u g
  Serial.println();                       // <---<<<<  D e b u g

Make this a function you can call.

[LarryD]

• When debugging State Machines, identify when we change state.

  //The states in your machine.
  enum State { IDLE, RUNNING, ERROR };             
  State machineState = IDLE;

  . . . 

  //When we change state, print the state we go to.     // <---<<<<  D e b u g
  void changeState(State newState)                      // <---<<<<  D e b u g
  {                                                     // <---<<<<  D e b u g
    if (newState != machineState)                       // <---<<<<  D e b u g
    {                                                   // <---<<<<  D e b u g
      Serial.print(F("State -> "));                     // <---<<<<  D e b u g
      Serial.println(newState);                         // <---<<<<  D e b u g
      machineState = newState;                          // <---<<<<  D e b u g
    }                                                   // <---<<<<  D e b u g
  }                                                     // <---<<<<  D e b u g

[alto777]

Yeth.

// The states in your machine.
enum State { IDLE, RUNNING, ERROR, NOSTATE};
char *tags[] = {"IDLE", "RUNNING", "ERROR", "WTF"};
       
State machineState = IDLE;



// somewhere, like before a switch on the state

{ // <-  your own scope
  static State printedState = NOSTATE;
  if (printedState != machineState) {
    Serial.print("machineState now = ");
    Serial.println(tags[machineState]);

    printedState = machineState;
  }
}

This debugging information makes the most sense when the state variable is actually the state the machinery is in.

a7

[LarryD]

[gcjr]

@LarryD

  //Print the line we are currently at:   // <---<<<<  D e b u g
  Serial.print("Line Executed = ");       // <---<<<<  D e b u g
  Serial.println(__LINE__);               // <---<<<<  D e b u g
  Serial.println();                       // <---<<<<  D e b u g

how abot a 1 line macro

setup: line   14
myFunc: line    8
myFunc: line    9

char s[90];

#define  Debug() sprintf (s, "%s: line %4d", __func__, __LINE__); \
                 Serial.println (s);

void myFunc () {
    Debug ();
    Debug ();
}

void setup () {
    Serial.begin (9600);
    Debug ();

    myFunc ();
}

void loop () {
}

[alto777]

Always always include a default case in your switch statements.

  switch (N) { 
// the cases you planned on

//... 

// the ones that happen that you did not plan on
  default :
    Serial.print("Rotten Denmark! No case ");
    Serial.println(N)
    for (; ; );
  }

This is part of planning that the code will have errors.

It will, won't it?

a7

[LarryD]

• Break points can be set where you can measure voltages etc.
  Use serial monitor, send R to continue.

   //Breakpoint loop that still accepts serial commands. // <---<<<<  D e b u g
   if (counter == 5000)                                  // <---<<<<  D e b u g
   {                                                     // <---<<<<  D e b u g
     Serial.println(F("Paused. Send 'R' to resume."));   // <---<<<<  D e b u g
     while(true)                                         // <---<<<<  D e b u g
     {                                                   // <---<<<<  D e b u g
       if (Serial.available())                           // <---<<<<  D e b u g
       {                                                 // <---<<<<  D e b u g
         if (Serial.read() == 'R')                       // <---<<<<  D e b u g
         {                                               // <---<<<<  D e b u g
           break;                                        // <---<<<<  D e b u g
         }                                               // <---<<<<  D e b u g
       }                                                 // <---<<<<  D e b u g
     }                                                   // <---<<<<  D e b u g
   }                                                     // <---<<<<  D e b u g

[gilshultz]

I tend to use pins to trace my code with either a scope or logic analyzer. starting a function I would set pin high and when exiting set the pin low. You can do this with several functions if you are tracing a conditional sequence. I also simply print a single character that is not normally in messages such as |. If you have color capability you can print it in any color you want such as red or green. The single character does not generally mess up the program's timing. I also place a 2 second delay in setup so if I get a serial flood the programmer can break it and get control.

[gcjr]

@LarryD

     Serial.println(F("Paused. Send 'R' to resume."));   // <---<<<<  D e b u g
     while(true)                                         // <---<<<<  D e b u g
     {                                                   // <---<<<<  D e b u g
       if (Serial.available())                           // <---<<<<  D e b u g
       {                                                 // <---<<<<  D e b u g
         if (Serial.read() == 'R')                       // <---<<<<  D e b u g
         {                                               // <---<<<<  D e b u g
           break;                                        // <---<<<<  D e b u g
         }                                               // <---<<<<  D e b u g
       }                                                 // <---<<<<  D e b u g
     } 

breakpoint use to mean halting a processor from running, either executing a halt instruction, see HLT or stopping the clock, allowing processor registers and memory to be examined with an emulator such as an HP 64000. (tedious)

debug code should typical be minmal (see bp())

output

05:58:23.257 -> loop
05:58:24.303 -> loop
05:58:24.352 -> bp:
05:58:31.503 ->  continue
05:58:31.503 -> loop
05:58:32.498 -> loop

// -----------------------------------------------------------------------------
void bp ()
{
    Serial.println ("bp:");
    while ('r' != Serial.read ())
        ;
    Serial.println (" continue");
}

// -----------------------------------------------------------------------------
const byte PinBut = A1;
unsigned long msec0;

void loop ()
{
    unsigned long msec = millis ();
    if (msec - msec0 >= 1000) {
        msec0 = msec;
        Serial.println ("loop");
    }

    if (LOW == digitalRead (PinBut))
        bp ();
}

// -----------------------------------------------------------------------------
void setup ()
{
    Serial.begin (9600);
    pinMode (PinBut, INPUT_PULLUP);
}

[robtillaart]

Common used debug techniques

• switch on all warnings and treat them as error (so solve them)
• print statements (as short as possible, preferably with timestamp)
• reading the code out loud
• logic analyzer, to see if and what signals are on the line
• oscilloscope, idem
• stripping code until it works again, time consuming unless you keep versions of code.
• remove all comments that do not explain the code. (comment = WHY, code = HOW ideally)
• check if input data still matches the functions written (e.g. a parameter became float).
• go back to design mode, put on the evil hat and think how one could break the code.
• do some math / processing with pencil and paper to understand what is needed.
  (a whiteboard is great for this)
• go for a walk, swim, run, shower, bed and restart later with a fresh head.

In some projects I use my heartBeat class to inform the user about the state by means of a LED. Mostly it is a "I am alive" message, but the class also supports blink patterns like
in a power on self test. Blinking LEDs consume far less CPU cycles than print statements.

• GitHub - RobTillaart/HeartBeat: Arduino library for a HeartBeat with frequency and duty cycle. · GitHub

Other uses are possible, in a project where a linear motor was drawing up to 20 Amps, I let the heartbeat indicate the current in Amps, faster was more current, not exact but good enough.

Finally, maybe the best trick - explain the bug to someone in as much detail as you can, what you expect, what happens etc. Often by telling the "bug" becomes visible.
(and just say thank you for listening).

[LarryD]

@robtillaart

explain the bug to someone in as much detail as you can, what you expect, what happens etc. Often by telling the "bug" becomes visible.
(and just say thank you for listening).

• I too can vouch that this works.
  My wife, with her eyes glassed over, has solved many of my problems without saying a word.

[robtillaart]

My wife, with her eyes glassed over, has solved many of my problems without saying a word

Like in all good marriages ;)

[oldcurmudgeon]

@robtillaart

Finally, maybe the best trick - explain the bug to someone in as much detail as you can, what you expect, what happens etc. Often by telling the "bug" becomes visible.
(and just say thank you for listening).

Or even a rubber duck https://rubberduckdebugging.com/