Serial Port Data Loss

Using Arduino Project Guidance
1

I have a simple setup with an Uno connected to two temperature probes via two AD8495 boards. The arduino code just polls the analog pins, averages 500 readings over 15 seconds, then outputs the calculated temperatures, differential temperature, voltages, and sample size to the serial monitor in JSON formatting. For the MOST part this works fine, but fairly often I get some lines where it stops halfway through a line and starts another or drops the first parts of one line.

Recently my serial monitor ends up looking something like this, with most lines showing up correctly, but several losing data.

{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1.326","sVolt":"1.330","rSamp":"500","sSamp":"500"}
{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1.326","sVolt":"1.330","rSamp":"500","sSamp":"500"}
{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1.326","sVolt":"1.330","rSamp":"500","sSamp":"500"}
{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1.326","sVolt":"1.330","rSamp":"500","sSamp":"500"}
{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1.326","sVolt":"1.330","rSamp":"500","sSamp":"500"}
":"63.7","sTemp":"63.7","dTemp":"0.0","rVolt":"1.328","sVolt":"1.328","rSamp":"500","sSamp":"500"}
{"rTemp":"63.1","sTemp":"64.2","dTemp":"1.1","rVolt":"1.326","sVolt":"1.330","rSamp":"500","sSamp":"500"}

Additionally, when this happens I tend to run into a lot of AVRDUDE Out of Sync or programmer not responding errors if I try and upload a sketch. The only I/O pins connected are A0 and A1, with a 5V pin feeding the two 8495s and a voltage divider connected to ARef and GND so I can better utilize the ADC for normal HVAC temperatures.

The Serial output is currently done via several print statements:

  Serial.print("{\"rTemp\":\"");
  Serial.print(String(returnTemperature,1));
  Serial.print("\",\"sTemp\":\"");
  Serial.print(String(supplyTemperature,1));
  Serial.print("\",\"dTemp\":\"");
  Serial.print(String(deltaTemp,1));
  Serial.print("\",\"rVolt\":\"");
  Serial.print(String(returnVoltage,3));
  Serial.print("\",\"sVolt\":\"");
  Serial.print(String(supplyVoltage,3));
  Serial.print("\",\"rSamp\":\"");
  Serial.print(sampleSize - returnErrors);
  Serial.print("\",\"sSamp\":\"");
  Serial.print(sampleSize - supplyErrors);
  Serial.println("\"}");

  delay(2);

But I ran into the same issue when using a StaticJsonDocument and serializeJson to output data to the serial port.

Right now I'm just guessing a bad board since it's some unbranded amazon-bought "UNO", but is there something else I should look at to ensure my serial data is clean and uninterrupted?

2 
#include <ArduinoJson.h>

#define RETURN_PIN A0          // set to ADC pin used
#define SUPPLY_PIN A1
#define AREF 1.576         // set to AREF, typically board voltage like 3.3 or 5.0
#define ADC_RESOLUTION 10  // set to ADC bit resolution, 10 is default 
#define BASE_VOLTAGE 1.250
#define MULTIPLIER 1 //set voltage divider mulitplier
#define OFFSET 0.010 //voltage offset for accurate temps


const int sampleSize = 500; //number of samples to go into each average
const int sampleRate = 20; //time between samples
const int minValue = 0;
const int maxValue = 1023;
const int maxErrors = 20;

float returnReading, returnVoltage, returnTemperature, supplyReading, supplyVoltage, supplyTemperature;
float returnArray[sampleSize], supplyArray[sampleSize];
float returnTotal, supplyTotal, deltaTemp;

float get_voltage(int raw_adc) {
  return raw_adc * (AREF / (pow(2, ADC_RESOLUTION)-1)) * MULTIPLIER;  
}

float get_temperature(float voltage) {
  return (voltage - BASE_VOLTAGE + OFFSET) / 0.005;
}

float get_temperature_f(float voltage) {
  return ((voltage - BASE_VOLTAGE + OFFSET) / 0.005 * 1.8) + 32;
}

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  analogReference(EXTERNAL);
  delay(2);
}

void loop() {

int returnErrors = 0;
float returnAverage = 0; 
float supplyAverage = 0;
int supplyErrors = 0;
float returnTotal = 0;
float supplyTotal = 0;
  for (int i = 0; i < sampleSize; i = i + 1) {
    returnReading = analogRead(RETURN_PIN);
    supplyReading = analogRead(SUPPLY_PIN);

    if (returnReading >= minValue) {
      if (returnReading <= maxValue) {
        returnTotal = returnTotal + returnReading;
      } else {
        returnErrors = returnErrors + 1;
      }
    } else {
      returnErrors = returnErrors + 1;
    }

     if (supplyReading >= minValue) {
      if (supplyReading <= maxValue) {
        supplyTotal = supplyTotal + supplyReading;
      } else {
        supplyErrors = supplyErrors + 1;
      }
    } else {
      supplyErrors = supplyErrors + 1;
    }
    delay(sampleRate);
  }

  returnAverage = returnTotal/(sampleSize - returnErrors);
  supplyAverage = supplyTotal/(sampleSize - supplyErrors);

  returnVoltage = get_voltage(returnAverage);
  supplyVoltage = get_voltage(supplyAverage);

  returnTemperature = get_temperature_f(returnVoltage);
  supplyTemperature = get_temperature_f(supplyVoltage);
  deltaTemp = supplyTemperature - returnTemperature;


  //StaticJsonDocument<300> jsonBuffer;
  //jsonBuffer["returnTemp"] = String(returnTemperature,1);
  //jsonBuffer["supplyTemp"] = String(supplyTemperature,1);
  //jsonBuffer["deltaTemp"] = String(deltaTemp,1);
  //jsonBuffer["returnVolt"] = String(returnVoltage,3);
  //jsonBuffer["supplyVolt"] = String(supplyVoltage,3);
  //jsonBuffer["returnSamples"] = sampleSize - returnErrors;
  //jsonBuffer["supplySamples"] = sampleSize - supplyErrors;

  //serializeJson(jsonBuffer, Serial);
  //Serial.println();
  //jsonBuffer.clear();

  Serial.print("{\"rTemp\":\"");
  Serial.print(String(returnTemperature,1));
  Serial.print("\",\"sTemp\":\"");
  Serial.print(String(supplyTemperature,1));
  Serial.print("\",\"dTemp\":\"");
  Serial.print(String(deltaTemp,1));
  Serial.print("\",\"rVolt\":\"");
  Serial.print(String(returnVoltage,3));
  Serial.print("\",\"sVolt\":\"");
  Serial.print(String(supplyVoltage,3));
  Serial.print("\",\"rSamp\":\"");
  Serial.print(sampleSize - returnErrors);
  Serial.print("\",\"sSamp\":\"");
  Serial.print(sampleSize - supplyErrors);
  Serial.println("\"}");

  delay(2);

}

For reference, here's the whole thing, complete with original serializeJson I was using commented out

3

A very common cause the your problem is having the Baud rate set way too low.

And I see you have a delay in there, which may also cause problems.

4

This would make me look closer at the board, the USB cable and the hub or port where you've plugged in.

I have no idea which is more likely or that any is the cause.

Maybe get a real UNO anyway. I've got a short list of things some clones I have aren't too good at, usually inconsequential but if I conduct a real experiment I use a real UNO.

You might be able to do some strictly software testing using the wokwi simulator, great for ruling out all kindsa pesky real life issues:

HTH

a7

5

How much ram does your sketch use?

6

I would at least get rid of the use of String


  Serial.print(F("{\"rTemp\":\""));
  Serial.print(returnTemperature, 1);
  Serial.print(F("\",\"sTemp\":\""));
  Serial.print(supplyTemperature, 1);
  Serial.print(F("\",\"dTemp\":\""));
  Serial.print(deltaTemp, 1);
  Serial.print(F("\",\"rVolt\":\""));
  Serial.print(returnVoltage, 3);
  Serial.print(F("\",\"sVolt\":\""));
  Serial.print(supplyVoltage, 3);
  Serial.print(F("\",\"rSamp\":\""));
  Serial.print(sampleSize - returnErrors);
  Serial.print(F("\",\"sSamp\":\""));
  Serial.print(sampleSize - supplyErrors);
  Serial.println(F("\"}"));

What type computer are you running the Serial monitor on?

7

My first thought as well. Compiled for an Uno, I did move the StaticJsonDocument<300> jsonBuffer to global memory and uncommented all json stuff.

Sketch uses 10082 bytes (31%) of program storage space. Maximum is 32256 bytes.
Global variables use 744 bytes (36%) of dynamic memory, leaving 1304 bytes for local variables. Maximum is 2048 bytes.

So that should be reasonably safe if there is not a serious amount of dynamic allocation at run time.

8

Yeah, having those errors definitely had me thinking more on the hardware/connection side. I found a genuine UNO that I had and uploaded the script and it's working exactly as expected, with the exception of one instance. I had the arduino IDE serial monitor open and started minicom on the same interface, and in that instance data was dropped but somehow it looked like half the data went to one monitor and half to the other. I don't know if that's expected, but when I closed the minicom session it returned to normal.

So far it seems like it was probably the arduino clone that was the primary culprit.

9

It's on a 10 year old desktop running Ubuntu in a closet.

That's a good point on removing string, it's usage there was a case of "the first google search had an example that worked using that syntax"

10

The baud rate I have set in the setup block is 115200, I wouldn't expect that to fall into the range of being considered too low, correct?

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  analogReference(EXTERNAL);
  delay(2);
}

Which delay causes concerns? The sampleRate delay?

const int sampleSize = 500; //number of samples to go into each average
const int sampleRate = 20; //time between samples

...

    delay(sampleRate);

What are the concerns to look for there? I know with using delay it prevents additional processing versus something like counting/tracking millis, but I didn't think that would be a major concern for the simplicity of this scenario. Would that be a potentially incorrect thought process?

11

It is for the amount of text data you are sending. Try replacing all the unnecessary text with a coded identification, such as a letter and a ":" for each type of data.

12

I don't have any problem with the code here, running on a Nano clone.

That should be ok, I have had problems with fast serial communications when running the serial monitor on a raspberry pi, but you do have lots of time between each line so there really should not be any problems.

The code the OP posted is only going to print one line of text every 10 seconds, because there is a loop taking 500 analog samples at 20mS intervals to get an average. There will be a small delay caused by the serial buffer, but the serial output is nowhere near saturated.

13

Ok, but the USB connection must be at the minimum speed for USB.

14

I definitely spoke too soon with calling the clone the culprit, the same issues have already popped up with the genuine UNO, including the "out of sync" and "Programmer not responding" errors when I attempt to upload a sketch. I have changed out the USB cable, swapped ports on the computer, and still get the same upload issues, UNLESS while changing ports the computer hops the arduino over to new port that's not being monitored (ttyACM1 from ttyACM0) and then the upload works fine. On ttyACM0 it fails on every upload with programmer out of sync, but I can still see the data output in the serial monitor. I guess maybe this is something on the computer side where however it's monitoring the serial port for HomeAssistant is causing communication and read issues?

15

So 115200 should still be reasonable as long as I limit the characters sent or is there a higher speed that would be recommended? 115200 is honestly just the one I've seen used the most so that's why I'm using it in this scenario

16

Can you either turn off the HomeAssistant or try another computer that does not have it running?

17

Yeah, when I pause the container HA is running in uploads magically work fine, and the serial monitor shows accurate data, and the moment I resume the container the data starts to get a bit wonky and sketch uploads start failing. So not a problem with the arduino it looks like

18

This and several other things you've said makes it look like you are trying to use the serial port for both the Arduino serial monitor and some other terminal emulator or program.

If so, don't do that.

You can listen to the Arduino <-> PC traffic, both sides, by using two additional serial inputs on your PC, however you do that. I use an old serial port analyser cable that breaks out TX and RX, and feed both those to RX inputs on terminal emulator windows. Presto poor man's serial lines monitor. Crude but effective.

You could use an Arduino with extra real serial UART ports. Software serial is not up to the high speeds you want on the comms, and as far as I know the relatively low demand of the serial monitor function can't be shifted off RX0 and TX1.

Closing the serial monitor should mean the serial at 0 and 1 should work fine be it with a terminal emulator or whatever other program wants to grab and use the port.

a7

19

Yeah, I think you're right and all this has been a self-caused problem by thinking I could have the serial monitor open and be accurate while HomeAssistant is constantly polling the serial port in the background and causing conflicts. I think I had left the IDE running with the serial monitor open at some point, which was probably when I started to get a ton of errors and I had no idea to connect the two....

It appears that with the serial monitor closed and just allowing HA to use the serial port in the background it is actually working as expected. And for issues with uploading sketches, I just have to pause the HA container while uploading, otherwise it will interrupt the programmer and cause it to fail.

Gotta learn the simple stuff some way, right? Thanks so much for the help everyone!

20

I'll be concerned if you start a new thread with the same problem! Glad YOU solved it!