UART new line detection

Hi all

I got the problem that my program don't detect every new line that I receive over UART.

My project;

I got 2 different PCB and I want to save all data which i receive over Uart to an SD card.

I'm using an Arduino DUE and a Datalogging shield from Adafruit.

From the first PCB (Atmega2560, 115200bd), which sends every 5 seconds some datas, it works perfectly.
With the second PCB (Atmega328pb, 9600bd), which sends every loop some information, it's not able to detect a new line. The result of that i got some uncompleted or mixed strings.

#include <SD.h>

// Date and time functions using a DS1307 RTC connected via I2C and Wire lib
#include <Wire.h>
#include "RTClib.h"

#if defined(ARDUINO_ARCH_SAMD)
// for Zero, output on USB Serial console, remove line below if using programming port to program the Zero!
#define Serial SerialUSB
#endif

RTC_PCF8523 rtc;

char daysOfTheWeek[7][12] = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"};

char inChar = 0;

char packetBuffer[100];   //incomming buffer

const int chipSelect = 10;

bool fullstringFlag1 = false;
bool fullstringFlag2 = false;


String uartString1 = "";
String uartString2 = "";

String completString1 = "";
String completString2 = "";
File ControllFile;
File BGFile;


void setup () {

#ifndef ESP8266
  while (!Serial); // for Leonardo/Micro/Zero
#endif

  Serial.begin(115200);
  Serial1.begin(115200);
  Serial2.begin(9600);

  uartString1.reserve(500);
  uartString2.reserve(500);

  if (!SD.begin(chipSelect))
  {
    Serial.println("Card reading failed");
    return;
  }
  Serial.println("Card initialized");


  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }

  if (! rtc.initialized()) {
    Serial.println("RTC is NOT running!");
    // following line sets the RTC to the date & time this sketch was compiled
    rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
    // This line sets the RTC with an explicit date & time, for example to set
    // January 21, 2014 at 3am you would call:
    // rtc.adjust(DateTime(2014, 1, 21, 3, 0, 0));
  }
  else
  {
    Serial.println("RTC initialized");
  }



}

void loop () {


  // Serial.println("loop");

  //ControllFile = SD.open("rtctest.txt", FILE_WRITE);
  //RTCfile = SD.open("RTCfile.txt", FILE_WRITE);

  DateTime now = rtc.now();


  /*while (Serial1.available() > 0) {
    DateTime now = rtc.now();
    Serial.println("read Serial1 1");
    inChar = (char)Serial1.read();
    uartString1 += inChar;

    if ( inChar == '\n') {  // New Line detection
      fullstringFlag1 = true;
      completString1 = uartString1;
      uartString1 = "";
      Serial.println("new line detected");
      Serial.println(uartString1);


      if (fullstringFlag1) {
        ControllFile = SD.open("CB.txt", FILE_WRITE);
        Serial.println("full string detected");
        if (ControllFile) {
          Serial.println("Write new data to sd card");
          Serial.println(completString1);

          ControllFile.print(now.year());
          ControllFile.print('/');
          ControllFile.print(now.month());
          ControllFile.print('/');
          ControllFile.print(now.day());
          ControllFile.print(" ");

          ControllFile.print(now.hour());
          ControllFile.print('/');
          ControllFile.print(now.minute());
          ControllFile.print('/');
          ControllFile.print(now.second());
          ControllFile.print(",");
          ControllFile.print(completString1);
          ControllFile.println("");
          ControllFile.close();
          completString1 = "";
          fullstringFlag1 = false;
        }// if ControllFile
        else {
          Serial.println("Conection lost");
          completString1 = "";
          fullstringFlag1 = false;
          ControllFile.close();
        }
      }// if fullstringFlag = true
    }//if c = \n
    }// while serial 1 available?*/

  if (Serial2.available())
  {
    while (Serial2.available() > 0) {
      // DateTime now = rtc.now();
      //Serial.println("read Serial1 2");
      inChar = (char)Serial2.read();
      uartString2 += inChar;

      if ( inChar == '\n') {  // New Line detection
        fullstringFlag2 = true;
       // completString2 = uartString2;
        //uartString2 = "";
        // Serial.println("new line detected");
        // Serial.println(uartString2);


        if (fullstringFlag2) {
          BGFile = SD.open("BG.txt", FILE_WRITE);
          //Serial.println("full string detected");
          if (BGFile) {
            //Serial.println("Write new data to sd card");
            //Serial.println(completString2);

            /*BGFile.print(now.day());
            BGFile.print(" ");

            BGFile.print(now.hour());
            BGFile.print('/');
            BGFile.print(now.minute());
            BGFile.print('/');
            BGFile.print(now.second());
            BGFile.print(",");*/
            BGFile.print(uartString2);
            BGFile.println("");
            BGFile.close();
            uartString2 = "";
            fullstringFlag2 = false;
          }// if BGFile
          else {
            Serial.println("Conection lost");
            uartString2 = "";
            fullstringFlag2 = false;
            BGFile.close();
          }
        }// if fullstringFlag = 1
      }//if c = \n
    }//while serial2 avaiable
  }//if serial2 available?
}// Loop

As more I comment out, as less this is happening.

For an example:

Sometimes I receive something like this;

bitSetting: ,1

U_elkoP ,132.81

U_elkoN ,

R

But it should be like;

bitSetting: ,1

U_elkoP ,39.72

U_elkoN ,42.41

Anyone some idea?

Thanks & best regards

I suggest that you receive the complete message before you try to do anything with the message. Have a look at how the code is organised in the 2nd example in Serial Input Basics. It just sets a flag (the variable newData) when it has received the message. Then a separate function deals with the message whenever it sees newData == true

If you are posting more code please remove all the commented-out lines to make it easier to read.

...R

Thank you for your reply, it works now much better.
I still got ± 1 line per second that ist faulty because of missing chars (not only new-line chars as I thought first).

It would be nice if you have another input, otherwise I can deal with this performance.

#include <SD.h>
#include <Wire.h>
#include "RTClib.h"

RTC_PCF8523 rtc;

File BGFile;
File ControllFile;

const byte numChars2 = 500;
const byte numChars1 = 500;

const int chipSelect = 10;

char receivedChars1[numChars1];    // an array to store the received data1
char receivedChars2[numChars2];   // an array to store the received data2

boolean newData1 = false;
boolean newData2 = false;

void setup() {
  Serial.begin(115200);
  Serial1.begin(115200);
  Serial2.begin(9600);

  Serial.println("<Arduino is ready>");

  if (!SD.begin(chipSelect))  {
    Serial.println("Card reading failed");
    return;
  }
  Serial.println("Card initialized");

  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }
  if (! rtc.initialized()) {
    Serial.println("RTC is NOT running!");  
    rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
  }
  else  {
    Serial.println("RTC initialized");
  }

}//setup

void loop() {
  recvWithEndMarker();
  saveNewData();
}


void recvWithEndMarker() {
  static byte ndx1 = 0;
  static byte ndx2 = 0;
  char endMarker = '\n';
  char rc1;
  char rc2;

  while (Serial1.available() > 0 && newData1 == false) {
    rc1 = Serial1.read();

    if (rc1 != endMarker) {
      receivedChars1[ndx1] = rc1;
      ndx1++;
      if (ndx1 >= numChars1) {
        ndx1 = numChars1 - 1;
      }
    }
    else {
      receivedChars1[ndx1] = '\0';
      ndx1 = 0;
      newData1 = true;
    }
  }

  while (Serial2.available() > 0 && newData2 == false) {
    rc2 = Serial2.read();

    if (rc2 != endMarker) {
      receivedChars2[ndx2] = rc2;
      ndx2++;
      if (ndx2 >= numChars2) {
        ndx2 = numChars2 - 1;
      }
    }
    else {
      receivedChars2[ndx2] = '\0';
      ndx2 = 0;
      newData2 = true;
    }
  }
}// End Void RECV


void saveNewData() {

  DateTime now = rtc.now();
  if (newData1 == true)  {
    ControllFile = SD.open("CB.csv", FILE_WRITE);
    ControllFile.print(now.day());
    ControllFile.print(" ");
    ControllFile.print(now.hour());
    ControllFile.print('/');
    ControllFile.print(now.minute());
    ControllFile.print('/');
    ControllFile.print(now.second());
    ControllFile.print(",");
    ControllFile.print(receivedChars1);
    ControllFile.println("");
    ControllFile.close();
    newData1 = false;
  }

  if (newData2 == true) {
    BGFile = SD.open("BG.txt", FILE_WRITE);
    BGFile.print(now.day());
    BGFile.print(" ");
    BGFile.print(now.hour());
    BGFile.print('/');
    BGFile.print(now.minute());
    BGFile.print('/');
    BGFile.print(now.second());
    BGFile.print(",");
    BGFile.print(receivedChars2);
    BGFile.println("");
    BGFile.close();
    newData2 = false;
  }
}//End Void saveNewData

Thank you & best regards

oduncu

How long (in characters) is each message that you receive and how often are messages sent?

You have set numChars to 500, but you can't have a number greater than 255 in a byte. You need to change numChars1 and 2 and ndx1 and 2 to the int type.

If a message is longer than 64 bytes then you must ensure that you call recvWithEndMarker() often enough so that the Serial Input Buffer gets cleared before it fills up. At 115200 baud 64 chars will arrive in about 5.5. millisecs.

AFAIK writing to an SDCard can be slow.

If it was my project I would have created two separate versions of recvWithEndMarker() - one for each serial connection, but your system may perform just as well.

...R

I have a similar setup as part of my personal Arduino RC airplane where the flight controller sends telemetry over wired serial to another Arduino for datalogging on an SD card. I use SerialTransfer.h and have had great success logging large amounts of telemetry (epoch time, lat, lon, pitch, roll, airspeed, altitude, and much more) at fairly high sample rates with no errors.

Maybe you'll find it useful for your application as well.

@Robin2

The messages are between 6 and 30 chars long and every 8/9 milliseconds I receive a new message.

For an example what I receive in a second;

16 16/38/1,BG_Ready , 0

16 16/38/1,BG_Ready , 0

16 16/38/1,unload POS,ON

16 16/38/2,unload NEG, ON

16 16/38/2,BG_Ready , 0

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,142.07

16 16/38/2,U_Elko_Neg,144.21

16 16/38/2,BG_Ready , 0

16 16/38/2,BG_Ready , 0

16 16/38/2,unload POS,ON

16 16/38/2,unload NEG, ON

16 16/38/2,BG_Ready , 0

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,141.22

16 16/38/2,U_Elko_Neg,142.77

16 16/38/2,TimeToBeReady,6374

16 16/38/2,BG_Ready , 1

16 16/38/2,Charge_Timer,581

16 16/38/2,TimeToBeReady,6374

16 16/38/2,BG_Ready , 1

16 16/38/2,Charge_Timer,600

16 16/38/2,unload POS,ON

16 16/38/2,unload NEG, ON

16 16/38/2,TimeToBeReady,6374

16 16/38/2,BG_Ready , 1

16 16/38/2,Charge_Timer,599

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,28.57

16 16/38/2,U_Elko_Neg,30.99

16 16/38/2,BG_Ready , 0

16 16/38/2,BG_Ready , 0

16 16/38/2,unload POS, OFF

16 16/38/2,unload NEG, OFF

16 16/38/2,BG_Ready , 0

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,35.86

16 16/38/2,U_Elko_Neg,38.31

16 16/38/2,BG_Ready , 0

16 16/38/2,BG_Ready , 0

16 16/38/2,unload POS, OFF

16 16/38/2,unload NEG, OFF

16 16/38/2,BG_Ready , 0

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,45.12

16 16/38/2,U_Elko_Neg,47.24

16 16/38/2,BG_Ready , 0

16 16/38/2,BG_Ready , 0

16 16/38/2,unload POS, OFF

16 16/38/2,unload NEG, OFF

16 16/38/2,BG_Ready , 0

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,54.42

16 16/38/2,U_Elko_Neg,56.48

16 16/38/2,BG_Ready , 0

16 16/38/2,BG_Ready , 0

16 16/38/2,unload POS, OFF

16 16/38/2,unload NEG, OFF

16 16/38/2,BG_Ready , 0

16 16/38/2,BIT_Setting,1

16 16/38/2,U_Elko_Pos,63.73

16 16/38/2,U_Elko_Neg,66.00

16 16/38/2,BG_Ready , 0

16 16/38/3,BG_Ready , 0

I changed now the code of the sending PCB, so that now every message is in minimum 12 chars long and now it works without some issues.
This will confirm your theory that the writing to the sd card is to slow?

@Power_Broker

So you collect all the datas on one arduino and send them to another arduino for datalogging on an SD card? Anyway thanks for your input, I will may find time for trying this library out

Thanks you guys! best regards

oduncu:
@Power_Broker

So you collect all the datas on one arduino and send them to another arduino for datalogging on an SD card? Anyway thanks for your input, I will may find time for trying this library out

Yup, that's how it works

oduncu:
The messages are between 6 and 30 chars long and every 8/9 milliseconds I receive a new message.

Do you really need to send messages that often? I suspect 4 or 5 times per second would be sufficient.

...R

Robin2:
Do you really need to send messages that often? I suspect 4 or 5 times per second would be sufficient.

...R

I got a main report which I send every 10 seconds. This report includes every non-critical informations.
For better imagine;
This sending PCB is an prototype flashgenerator and with that I need to to an LTtest. On that PCB there are 24pcs of 2'000uF Elkos which I can load up to 720V (12pcs from 360V - 0V & 12pcs from 0V - -360V) and the loading time is between 1.5 and 6 seconds. So if something doesn't work as it should, it could be pretty dangerous;) For that reason I need so much information as I can for reproducing any failures.

best regards
oduncu

oduncu:
So if something doesn't work as it should, it could be pretty dangerous;)

You should not be relying on a communications link for safety issues. They should be handled on the microprocessor where they arise. Then, at its leisure it can send a message to HQ saying "Sorry, I'v had to shut it down"

...R

I have some emergency shutdown function implemented which tell me why he shut off. With this received information I do nothing, just logging, monitoring them and analyse them afterwards.

But in case the uC latch up and/or the hole thing explode (shouldn't happening anymore but you never know) I'm able see what happened before.

oduncu:
With this received information I do nothing, just logging, monitoring them and analyse them afterwards.

Then there is no need to send data at 8msec intervals. 800msec might be sufficient (or 400).

If you want details of what leads up to a crash you could have the micro-processor keep updating a circular array of data. Then if there is a crash it could send that data (at its leisure) to HQ. The equivalent of the flight-recorder on an airplane.

...R

For this case I'm with you.

But I also have to compare the charge-cycles of the elkos and some others stuff.
eg.

  • Are both rails loading equivalent to each other.
  • How does the charge cycle change over time? eg. Are they similar after 100k cycles?
    For that I need as much data as possible to create a charging "curve".

So this data I could also write to an array and send them all after one cycle, but wont this take them to much time at once? I always need to control the voltage niveau of the capacitors to adjust the loading frequency and prevent overloading and other stuff.
btw, I can't increase the bd, because it's an isolated UART.

best regards

Please provide a lot more information about what you are trying to achieve.

So far you have been asking questions about how YOU think the problem should be tackled without providing all the details of what needs to happen. It is better to start by setting out the requirement without worrying about how it might be implemented. There may be a much simpler way to achieve what is needed.

...R