Serial communications are blocking interrupts and hanging my program

Where to start... I have code that has enabled all 3 timers on the arduino (0, 1, and 2) for Fast 8-bit PWM mode. As of right now, the Output Compare Registers are set to a static value (max 255) with the pre-scaler set as the clock speed. The interrupt routines run fine in my testing and perform read values from pins A0-A5, which will be stored into global variables (declared as volatile) which will be accessed later in the program.

Globals.h

extern volatile unsigned int PIN_A0_READ;
extern volatile unsigned int PIN_A1_READ;
extern volatile unsigned int PIN_A2_READ;
extern volatile unsigned int PIN_A3_READ;
extern volatile unsigned int PIN_A4_READ;
extern volatile unsigned int PIN_A5_READ;

Thermistor.h (Constructor, notice how the read value is passed by reference to the instantiated object. Thermistor class methods only perform reads and DO NOT WRITE!!!)

Thermistor(unsigned int *analogRead, int pinAddress, long int nominalResistance, float R2, float lineVoltage,
           int beta = 3950, int nominalTemperature = 25) :
        adcValue(analogRead), logicalPinAddress(pinAddress), nominalResistance(nominalResistance),
        lineVoltage(lineVoltage), r2(R2), beta(beta), nominalTemperature(nominalTemperature){};

All seems well at first; I can print the global values without a problem in the main loop and I see that they are being read and written correctly over the Serial monitor. However, when I try printing more statistics over the Serial which are longer in size, it appears to stop reading the values from the ADC and completely hang the program toward the end of the buffer.

I've done some research on this topic and it seems as though the print() and println() functions from the Serial are supposed to block Interrupts by temporarily disabling them. This may explain why the ADC is not updating it's values.. perhaps the print statements occur first and the ISR does not have a chance to read the sensor value beforehand. If this were the case, it would not take very long to print a series of strings when the interrupts are disabled. And yet-- it takes a good 10 seconds or so to print out all of my statistics with a max baud rate (115200). If I disable all ISR's manually by commenting out their definitions, then print the same information, it prints things to the console almost instantaneously, which is indicating that the ISR's are in fact-- being re-enabled and triggered intermediately between the various Serial print calls. What's interesting is that the print behavior seems to work fine up until the very last few lines which is where the main loop stalls. Additionally, why are the global variables no longer being updated if functioning correctly?

The Arduino API should automatically flush the output buffer when it gets filled. But Strings are large in size, and I actually encountered some memory fragmentation issues prior when printing very large strings all at once. To avoid this as much as possible, I have a small check implemented in my printConfig() method which forces the buffer to flush() if the Arduino does not have less than 32 bytes available on the heap. This implies that I am only really trying to print 32 bytes or less to the Serial port at a time in this function, and performing a mem-check and flushing in between only when it is necessary. This seems to work fairly reliably and it does not stall the program when only trying to print and call a single getter method from the class.

I do not really know the cause of this weird behavior. It only stalls when trying to iterate between multiple Thermistor class objects and invoking the printConfig() method. The stuff that does print out, does not have an updated read value from the ADC.

What is going on here?

Thermistor.cpp (Note: the ArduinoSTL is a library ports over the std namespace. Changing this to the print() println() methods from the Arduino.h API makes no difference at all in the buggy behavior)

void Thermistor::printConfig()
{

    for (int j = 0; j < 10; ++j)
    {
        // Check to see if SRAM memory available is less than 32 bytes to load string.
        if (freeMemory() < 12 + 20) // Largest value size is 12 (long double). Prompt size is < 20 bytes.
        {
            // If it is, flush to the Serial buffer to free up space.
            std::cout.flush();
        }
        else
        {
            switch(j)
            {
                case 0:
                    std::cout << "Logical Pin Address: " << getLogicalPinAddress() << " ";
                    break;
                case 1:
                    std::cout << "Analog Read: " << getAdcValue() << " ";
                    break;
                case 2:
                    std::cout << "Nominal Resistance: " << getNominalResistance() << " ";
                    break;
                case 3:
                    std::cout << "Beta Value: " << getBeta() << " ";
                    break;
                case 4:
                    std::cout << "Supply Voltage: " << getLineVoltage() << " ";
                    break;
                case 5:
                    std::cout << "Voltage Output: " << getVo() << " ";
                    break;
                case 6:
                    std::cout << "R1: " << getR1() << " ";
                    break;
                case 7:
                    std::cout << "R2: " << getR2() << " ";
                    break;
                case 8:
                    std::cout << "Temperature (C): " << getTemperatureCelsius() << " ";
                    break;
                case 9:
                    std::cout << "Temperature (F): " << getTemperatureFahrenheit() << " ";
                    break;
            }

        }
        std::cout << '\n';
    }
    std::cout.flush();
}

main.cpp

// Iterate through vector of Thermistors
for (int i = 0; i < sensors.size(), ++i;){
    Thermistor * T;     
    T = &sensors.at(i); // get reference to object inside.
    T->printConfig();   // print out the Thermistor Statistics.
    std::cout << std::endl; // newline betweeen each probe for readability.
}

Serial Monitor results

Logical Pin Address: 15 
Analog Read: 0 
Nominal Resistance: 20000 
Beta Value: 3950 
Supply Voltage: 5.00 
Voltage Output: 0.00 
R1: inf 
R2: 20000.00 
Temperature (C): inf 
Temperature (F): 0.00 

Logical Pin Address: 16 
Analog Read: 0 
Nominal Resistance: 20000 
Beta Value: 3950 
Supply Voltage: 5.00 
Voltage Output: 0.00 
R1: inf 
R2: 20000.00 
Temperature (C): inf 
Temperature (F): ovf 

Logical Pin Address: 17 
Analog Read: 0 
Nominal Resistance: 20000 
Beta Value: 3950 
Supply Voltage: 5.00 
Voltage Output: 0.00 
R1: inf 
R2: 20000.00 
Temperature (C): inf 
Temperature (F): ovf 

Logical Pin Address: 18 
Analog Read: 0 
Nominal Resistance: 20000 
Beta Value: 3950 
Supply Voltage: 5.00 
Voltage Output: 0.00 
R1: inf 
R2: 20000.00 
Temperature (C): inf 
Temperature (F): ovf 

Logical Pin Address: 19 
Analog Read: 0 
Nominal Resistance: 20000 
Beta Value: 3950 
Supply Voltage: 5.00 
Voltage Output: 0.00 
R1: inf 
R2: 20000.00 
Temperature (C): inf 
Temperature (F): 0.00
1 Like

aspen1135:
Where to start...

You need to post your complete program and also tell us what Arduino board you are using.

Have you tried using a higher baud rate?

...R

Robin2:
You need to post your complete program and also tell us what Arduino board you are using.

Even better post an MRE that does nothing more that compiles and demonstrates the problem (and only the problem). Leave out any extraneous clutter code that's not involved with the issue at hand.

When code acts funny I recommend:

  • Go to Preferences and set Compiler warnings to "All"
  • Compile your sketch.
  • Read and fix (or completely understand why it doesn't need fixing) EVERY warning the compiler provides.

I would just say that at least for the 328P there should be no need for print() to disable interrupts. The USART has its own dedicated interrupt vectors, and the module should finish sending or receiving a byte independently of what's going on elsewhere. But I've never looked at the print() code, so perhaps it's doing something like that.

I've done some research on this topic and it seems as though the print() and println() functions from the Serial are supposed to block Interrupts by temporarily disabling them.

That's news to me. Can you post a link to that information?

ShermanP:
I would just say that at least for the 328P there should be no need for print() to disable interrupts. The USART has its own dedicated interrupt vectors, and the module should finish sending or receiving a byte independently of what's going on elsewhere. But I've never looked at the print() code, so perhaps it's doing something like that.

That's news to me. Can you post a link to that information?

I had to go back and find some of the posts I found from google. Upon a second inspection, I may have miss-interpreted some of the information :confused: It seems that the print() statements pushes information to a buffer and it is flushed asynchronously via a register that is then disabled after it has completed*.* I somehow was interpreted this as the print statement first disabling all other timer interrupts, flushing, then re-enabling all interrupts instead. I supposed it is an understandable mistake, as there is some discussion on how Serial and timer interrupts interact, and one solution proposed suggested interrupt guards.

but this also makes me question the behavior of it's asynchronous behavior: when a flush is invoked, does it push a single byte at a time while leaving the Timer interrupts enabled? Does this potentially interfere with the Timers as it consumes additional cycles to do so?

Another user encountered a similar bug which he traced down to an issue with the Arduino API itself. However, his situation was a little different, as he was not using a PWM mode for Timer 2 like I am, and he was running very timing-critical code. I'm sure this bug has been patched by now since the post was back in 2012.

For this topic, I tried to show only the code in my project which pertained to this issue. What I have shown so far is actually most of my project's code in general. I'm working on cleaning up some things up (mostly commented out gargen from previous testing) and then I'll upload the thing to Github so you can check it out. It's a simple project which uses an Arduino nano as a temperature monitoring and fan speed controller for my 3D printer-- there's not a whole lot of complicated stuff to it.

btw, if it has not been said already, the project is running on an Arduino Nano using a max baud rate of 115200.

   if (freeMemory() < 12 + 20) // Largest value size is 12 (long double). Prompt size is < 20 bytes.

{
            // If it is, flush to the Serial buffer to free up space.
            std::cout.flush();
        }

Well, this doesn't do anything useful. At least, not if the other comment about "Using STL instead of print doesn't matter" is true. The Arduino core libraries (excepting Strings) doesn't do any dynamic memory allocation at all.

std::cout << "Logical Pin Address: " << getLogicalPinAddress() << " ";

Exactly which ArduinoSTL are you using?
does it support F("string") ? That would save you a bunch of RAM. Right now, all your literal strings are using RAM.

What I have shown so far is actually most of my project's code

Perhaps. But it's not in a state where we could download, compile, and analyze the object file to see whether it's doing dynamic allocation that it shouldn't...

Alright. I created a repository that you can download and compile from. I am using Jetbrains Clion IDE along with the PlatformIO plugin for it. If you are using the same, you can simply do this:

  1. Download the repository from github.
  2. Open the project in Platformio
  3. Go to Tools->PlatformIO->Re-Init.

PlatformIO should automatically create the Cmake environment from there.

Hit run and it will upload to the board. Note that the PlatformIO enviroment settings are configured for an Arduino Nano which is what this project is built around.

or you can use the Arduino IDE and build/upload from there. I am not too sure if you can just change the main.cpp extension to .ino for the IDE to recognize it, nor do I know how to build and compile a project structure that includes multiple headers and source files. I do not really like the Arduino IDE. Maybe you can import it as a library...?

Here is the repo:

aspen1135:
I had to go back and find some of the posts I found from google. Upon a second inspection, I may have miss-interpreted some of the information :confused: It seems that the print() statements pushes information to a buffer and it is flushed asynchronously via a register that is then disabled after it has completed*.* I somehow was interpreted this as the print statement first disabling all other timer interrupts, flushing, then re-enabling all interrupts instead. I supposed it is an understandable mistake, as there is some discussion on how Serial and timer interrupts interact, and one solution proposed suggested interrupt guards.

You don't need to google anything, you have the source code to the entire Arduino system in your
computer as its an open-source system... just grep(*) for the relevant function names in your Arduino
install directory and read the code in question for yourself.

Or you can go to the Arduino github repo and peruse the source there - for instance HardwareSerial is
here: ArduinoCore-avr/cores/arduino/HardwareSerial.cpp at master · arduino/ArduinoCore-avr · GitHub

As far as I see the print code may wait for interrupts to happen, its wouldn't be any use to block them
as then the Serial system would be jammed up.

(*) grep - grep - Wikipedia

aspen1135:
Alright. I created a repository that you can download and compile from.

Why not make it easy for people to help you by posting the program here.

If the program is so big that it needs a repository I am certainly too lazy to look at it. Make a short program that illustrates the problem as recommended in Reply #2 and post that.

...R

for (int i = 0; i < sensors.size(), ++i;){

Did you really intend to increment 'i' at the TOP of the loop?!? A more normal 'for' loop is:

for (int i = 0; i < sensors.size(); ++i){

I think that bizarre use of the 'for' loop may be causing you to operate on 'sensors' entries 1 through 6 instead of 0 through 5... and there is no entry 6.

You appear to be running all six timer interrupts at 62.5 kHz and in each ISR you are calling analogRead(). I don't think analogRead() can run at 62.5 kHz, let alone at six times that (375 kHz)! I suspect you are never leaving a time ISR without having all 6 ISR's queued for service.

The ADC is rated for a maximum of 15,000 samples per second at full resolution. This is done with a 200 kHz clock but the available clock prescale factors do not include 80. The next higher available prescale is 128 so the highest full-resolution clock rate you can get is 125 kHz. At 13 clocks per sample that comes to a sample rate of just under 10 ksps for ALL CHANNELS COMBINED. About 1.6 ksps if you are sampling all six channels in sequence.

If you don't need all 10 bits of resolution you can crank the prescale down to 16 (1 MHz clock) and get a sample rate of about 77 ksps. Your absolute accuracy drops from 2 LSB to 4.5 LSB but if you throw away the bottom two or three bits that should be close enough. That would get you over 10 ksps per channel.

Running six clock interrupts is likely to be a bad idea. It makes MUCH more sense to put the ADC in free-running mode and have it interrupt whenever it has a sample ready. That way you don't have to wait for the ADC setup between sample which would further reduce your sample rate.

Of course, there is no need to do an analogRead() before you need the thermistor temperature. It makes no sense to try to read 62000 samples per second per channel if you don't need 62000 temperature readings per second. Just have the Thermistor object do an analogRead() when it needs the data.

johnwasser:

for (int i = 0; i < sensors.size(), ++i;){

Did you really intend to increment 'i' at the TOP of the loop?!? A more normal 'for' loop is:

This was the problem all along. Damn... ! :confused:

Thanks for pointing it out! That fixed it. Doing the analog reads in the ISR was just temporary for proof of concept. My initial idea was just to perform a read before doing a write on each PWM channel since the fan speed directly correlates to how hot the the thermistor is. I didn't really think about how it would clog up the call stack too much.

After your explanation I agree that is probably not the most optimal approach.. When I was reading the timer's datasheet I noticed there was some input modes as well for at least one of the timers but I hadn't played around with them yet. I also probably don't need the full resolution on the ADC either for such sensors. They will only be used around 20- 80 Celsius and can be accurate to (+-) one or two degrees for such a project.

After deployment, there really probably wont be anything printing over the Serial port anyway but it is useful to have for debugging. I'm still an amateur to the Arduino, but hey-- you gotta learn somehow! : )