Serial.Println causing freeze During TFLM (TFLite Micro) Invoke on Nano 33 BLE

I'm sorry for such an amateur or unstructured question since this is my very first question on the community but I'm frustrated and been dealing with this issue for almost a year and couldn't really find a reference or keyword for similar problem.

Let's start and please bare with me. I'm doing a machine learning implementation on Arduino Nano 33 BLE. What I've done :

  1. Load the TFLM model to Nano without problem
  2. Have the model to inference successfully

The problem is, once I try to "log" the code by putting Serial.println (or .print), it becomes stuck in the invoke process. Some combination of the character to be printed inside Serial.print("----"); matters.

Example: as you know, the universal invoke line in TFLM is

TfLiteStatus invoke_status = interpreter->Invoke();

If I run my whole code with that, it didn't work. However, if I put a line before and it looks like:

Serial.println("=============Begin Processing ...=============");
TfLiteStatus invoke_status = interpreter->Invoke();

It works just fine. But, if I change the character inside the println, it will also become stuck again and not responding before it does the inference. This is just one example of the weird behavior I found precisely.

In another instance, after experimenting with a lot of Serial.print, I sometimes notice that it will be error when there is a Serial.print line before the invoke but not after the invoke. In another instance, when I change the

Serial.println("=============Begin Processing ...=============");

into

Serial.println("Test");

then it will break the program and freeze the code during the invoke part. It also sometimes breaks when I add more Serial.println.

It does sound ridiculous and I need to convince myself that I'm not crazy because it doesn't make sense for me as well. Therefore, I've been using some Serial.print combination that works and just leave it as is.

However, now I need to modify and upgrade my implementation and it will cause more problem down the line cause I don't know what works and what breaks that.

If you have any insight or keywords on what's going on, please help me.

I'm trying to get insight on what the error is and what causing it. I tried hundreds combination of Serial.print() before and after the invoke line but sometimes it works and do the invoke, sometimes it's not.

One thing to note is it is consistent, so once the combination works, it will always work. However, changing one character inside the print might freeze the program.

UPDATE:
After hours of trying what works or not, I somehow can conclude that it seems to be the number of character to be printed before the invoke matters.

The test I tried were:

  • Pick one of the SerialPrint text that works
Serial.print("AllocateTensors success!\n");
  • Change that into these line and it doesn't work
  Serial.print("AllocateTensorssuccess!\n");
  • Instead of removing the space, I change it with another character to make sure the number of characters are the same and it works
Serial.print("AllocateTensorssuccessX!\n");

The question now is how do we define what works and what not?

I have no experience with your board or the library.

Your description shows the typical symptoms of running out of memory.

1 Like

Thank you for the idea. But is there any certain way to make sure that it is a run out of memory problem?

Since this issue sometimes can be solved by adding more Serial.println ("other text"); on other line, I don't think it's a memory issue unless my current understanding is not correct.
If that's a memory issue, is it possible adding more command to print can solve the memory issue?

I should have been more accurate; instead of "running out of memory", it should indeed have been "a memory issue". You might be overwriting something somewhere which results in e.g. a reset. When you modify your code and add e.g. a Serial.print, the layout of variables changes and you no longer overwrite the same memory but something else what can be less obvious (e.g. a wrong result in a calculation). It might also be that you're running out of memory resulting in unpredictable results.

You can search for SRAM Memory Measurement in https://docs.arduino.cc/learn/programming/memory-guide/. You need to call display_freeram() in critical places; the first one would be in the beginning of loop(). If the amount reported becomes smaller and smaller, it indicates that your sketch is eating memory and will eventually crash. If it doesn't become smaller, start adding the call of display_freeram() to the beginning and end of functions that you use (this might eventually have to include functions in the libraries that you use if you have to get to the bottom of it).

1 Like

I appreciate the prompt response and your explanation! I will try checking the RAM shortly.

In the meantime, I would like to share my new findings. When I tried to add another Serial.print, I found this behavior:

Try printing these string lines below and some work, some doesn't work but it has pattern that correlate with the number of character inside the Serial.print

"Inputdata for TFLM:" works -> 19 char
"Input_data for TFLM:" 20 char -> doesn't work
"Input daataa for TFLM:" -> 22 char -> doesn't work
"Input dataaaa For TFLM:" -> 23 char -> works again
"Input dataaaaa For TFLM:" -> 24 char -> doesn't work
"Input dataaaaaaaa For TFLM:"-> 27 char -> works

From those trial, it shows the program works every 4 character in the Serial.print. Does this maybe have useful value to pinpoint alternative root cause of this?

You’re on the right track…
The tricky part - especially on a small memory processor - is the difference between ‘compile time’, and ‘run time’ errors - particularly regarding memory.

The compiler has almost no awareness of stack & heap usage, even in a ‘perfectly’ written program

Pointers, arrays and large calling structures,, along with recursion can eat available memory at an alarming rate.. unmonitored, so, overwrites and overflows are quite easy to occur if you’re not panning and keeping track of your code and data structures.

More complex processors allow you to place traps or breakpoints around code segments that you’re investigating.

1 Like

Thank you for the advice! At my current knowledge, I have no idea about those 'compile time', 'run time', 'stack & heap usage' so I will start learning what are those and how it will affect the C++ code and program because I really want to understand what's happening.

Also, as for overwrites and overflows, is there any way to monitor if it's happening? I would normally just log every step using Serial.print but now since it might cause error itself, I'm not sure how to debug or troubleshoot those.

It’s interesting that you phrase your answer this way,…

Us old guys and girls learnt to code before there were widely available compilers, hence , using assemblers and hand coding, we learned more about memory user than is reasonably expected of an application programmer.

System programmers need to understand more, but they still stuff it up all too often. Read as software bloat… a complex program that /could/ run in a few hundred K, is allocated two three times as much space as needed to reduce the occurrence of memory crashes, yet.. left long enough, it will still crash anyway!

There are small utility functions to print available SRAM, but they’re only useful about half the time, because ,emory is shuffled around in the background without your code doing anything, The only real solution is a more sophisticated processor with hardware debugging facilities built in.

1 Like

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.