Reading Serial output of multiple sensors of the same type

Projects Programming
21

I put my comments in bold under your comments.

drmpf:
Hi,
A couple of points. For Mega, the recommended software serial appears to be AltSoftSerial library.

You can mix BufferedOutput on one output with non-buffered outputs no problem

No problem using softserial apart from that it will block and you will miss some sensor reading.
On the other hand you can buffer its output as well

So do you mean, I can avoid these missing sensor values, when I use your library properly? Even with SoftwareSerial?

For your 3min json output you can createBufferedOutput(jsonOut, 120, BLOCK_IF_FULL);
Adjust the 120 to be large enough for your whole message.
Then in the setup connect it to your softserial
jsonOut.connect( softserial )
and add in the loop() jsonOut.nextByteOut();

For reading timestamps, yes using available() and read() works, but you have to be careful to avoid coding errors, buffer overflow, missing terminating nulls. Once you have the data in a char[] you can wrap it in a SafeString to safely split it into parts. See the SafeString library example sketches and the SafeString tutorial
OR you can put a '\n' (newline) on the end of you timstamp and use readUntilToken again to collect the line for processing. Either way will work. Using readUntilToken avoids the low level byte by byte handling code.

I do it at the moment like that: From my other Microcontroller device with internet capability, I send a timestamp string which looks like that <2020,12,24,13,30,35> to my Arduino, so I update my RTC. I use a DS3231 and I play to do that like once a week so the time on the RTC is always correct. The RTC has drifting issues according to the datasheet. Guess I will look into readUntilToken then.

As noted in the tutoral, if you call Serial directly it will 1) by pass the buffered output and inject chars in the middle of some other message and 2) will block your loop. Not recommended, so use output.print() instead.
You can change the bufferedOutput to BLOCK_IF_FULL to get ALL the output but it will block you loop. OR you can use clearSpace() and protect() for special messages OR you can call output.flush() first to completely clear the buffered output and then output.print() and then output.flush()
Just changing to BLOCK_IF_FULL is often the easiest. You get a bigger output buffer but otherwise is works just like Serial.print()

I will probably use the alternative, which will not block, which means clear.space and protect. I will try to reduce my serial outputs for debugging to a minimum, so I dont have to write too many functions.

It would probably be a good idea to just code up the timestamp / json code in a separate sketch and add a loopTimer to see what is happening and get that code running correctly before integrating it into the main sketch.

Good idea. I will try to code this modules separately based on your examples and library. Thanks for the suggestion.

You can post that sketch here if you need help.

Thank you!

Thank you for all your detailed help. I will try to cook something up. :slight_smile:

22

For debugging, one of the tips in Serial Text I/O for the Real World is to abbreviate the text
That is send "L:" instead of "Loop:" so more debugging can be output without dropping any

So do you mean, I can avoid these missing sensor values, when I use your library properly? Even with SoftwareSerial?
If you add a BufferedOutput for the SoftwareSerial then the only problem will be the SoftwareSerial/buffering/loop processing time that increases with more code. No delays due to SoftwareSerial blocking.

23

Hi,

thanks again for the input. :slight_smile:

I've written some separate Sketch for the json data transmission part (Arduino -> Modem).

It only works though, when I use jsonOut.connect(Serial). When I use jsonOut.connect(ModemPort) the Code can't be compiled.

The Code:

// comment this line out to use real sensor data
#define TEST_DATA
#include "SafeString.h"  // download from Arduino Library manager
#include "SafeStringStream.h" // included in V2.0.7 of SafeString
#include "BufferedOutput.h" // included in V2.0.7 of SafeString
#include "loopTimer.h" // install loopTimer from zip file
#include "SoftwareSerial.h"   // https://www.arduino.cc/en/Reference/SoftwareSerial communication between Arduino and Module

///  from https://www.forward.com.au/pfod/ArduinoProgramming/RealTimeArduino/index.html

createBufferedOutput(jsonOut, 400, BLOCK_IF_FULL);

// Softwareserial objects
SoftwareSerial ModemPort(11, 12);    // RX Pin, TX Pin

const int Number = 1;  

// Simulated measurement intervall
unsigned long startMillis;  
const unsigned long measurementPeriod = 2000;

#ifdef TEST_DATA // replace measurements with test data
float Bat1 = 3.64; float Bat2 = 3.67;

// 3 minute means of measurements
float t1 = 20.53; float h1 = 66.16;
float t2 = 20.92; float h2 = 56.05;

float o31 = 23; float o31_ug = 46;
float o31_t = 20; float o31_h = 65;
float no21 = 0; float no21_ug = 0;
float no21_t = 20; float no21_h = 64;
float co1 = 6285; float co1_mg = 6;
float co1_t = 20; float co1_h = 65;
float o32 = 8883; float no22 = 16527;
float co2 = 2331; float co3 = 22000;
float PM1 = 3.85 ; float PM2 = 4.07;
float PM4 = 4.07; float PM10 = 4.07;

// timestamp in measurement string
int year = 2020;
byte month = 11;
byte day = 12;
byte hour = 20;
byte minute = 45;


#else  // use real inputs from sensors

float Bat1; float Bat2;

// 3 minute means of measurements
float t1; float h1;
float t2; float h2;

float o31; float o31_ug;
float o31_t; float o31_h;
float no21; float no21_ug;
float no21_t; float no21_h;
float co1; float co1_mg;
float co1_t; float co1_h;
float o32; float no22; float co2;
float PM1; float PM2;
float PM4; float PM10;

// timestamp in measurement string
int year;
byte month;
byte day;
byte hour;
byte minute;

#endif


//============

void setup() {
  Serial.begin(9600);
  ModemPort.begin(9600);
  
  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }
  
  Serial.println();
  Serial.println(F("Demo for sending a json measurement string to a Modem."));
  Serial.println(F("Based on the SafeString V2.0.8+ library."));
  Serial.println();
  // Uncomment, if you want to use SoftwareSerial
  //jsonOut.connect(ModemPort);
  //SafeString::setOutput(ModemPort); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!
  // Comment, if you dont want to test via Serial console
  jsonOut.connect(Serial);
  SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!

}

//============

void loop() {
   jsonOut.nextByteOut(); // output next buffered chars. Need to call this at least once each loop
   //loopTimer.check(jsonOut);
   if (millis() - startMillis >= measurementPeriod) {
    startMillis = millis();
    SendDataModem(t1, h1, t2, h2, o31, o31_ug, o31_t, o31_h, no21, no21_ug, no21_t, no21_h, co1, co1_mg, co1_t, co1_h, o32, no22, co2, co3, PM1, PM2, PM4, PM10, year, month, day, hour, minute, Bat1, Bat2);
   }
}


//============
void SendDataModem(float t1, float h1, float t2, float h2, float Oz1, float Oz1_ug, float Oz1_temp, float Oz1_hum, float NitDio1, float NitDio1_ug, float NitDio1_temp, float NitDio1_hum, float CarMon1, float CarMon1_mg, float CarMon1_temp, float CarMon1_hum, int16_t Oz2, int16_t NitDio2, int16_t CarMon2, int16_t CarMon3, float PartMat1, float PartMat2, float PartMat4, float PartMat10, int yyyy, byte mon, byte dd, byte hh, byte mm, float AdaBat, float ModemBat) {
  cSF(jsonMsg, 400);
  jsonMsg.print(F("{\"Id\":")); jsonMsg.print(Number); jsonMsg.print(F(","));
  jsonMsg.print(F("\"Y\":")); jsonMsg.print(yyyy); jsonMsg.print(F(","));
  jsonMsg.print(F("\"M\":")); jsonMsg.print(mon); jsonMsg.print(F(","));
  jsonMsg.print(F("\"D\":")); jsonMsg.print(dd); jsonMsg.print(F(","));
  jsonMsg.print(F("\"Hr\":")); jsonMsg.print(hh); jsonMsg.print(F(","));
  jsonMsg.print(F("\"Mi\":")); jsonMsg.print(mm); jsonMsg.print(F(","));
  jsonMsg.print(F("\"B1\":")); jsonMsg.print(AdaBat); jsonMsg.print(F(","));
  jsonMsg.print(F("\"B2\":")); jsonMsg.print(ModemBat); jsonMsg.print(F(","));
  jsonMsg.print(F("\"T1\":")); jsonMsg.print(t1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"H1\":")); jsonMsg.print(h1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"T2\":")); jsonMsg.print(t2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"H2\":")); jsonMsg.print(h2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1\":")); jsonMsg.print(Oz1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1g\":")); jsonMsg.print(Oz1_ug); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1t\":")); jsonMsg.print(Oz1_temp); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1h\":")); jsonMsg.print(Oz1_hum); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1\":")); jsonMsg.print(NitDio1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1g\":")); jsonMsg.print(NitDio1_ug); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1t\":")); jsonMsg.print(NitDio1_temp); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1h\":")); jsonMsg.print(NitDio1_hum); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1\":")); jsonMsg.print(CarMon1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1g\":")); jsonMsg.print(CarMon1_mg); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1t\":")); jsonMsg.print(CarMon1_temp); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1h\":")); jsonMsg.print(CarMon1_hum); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O2\":")); jsonMsg.print(Oz2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N2\":")); jsonMsg.print(NitDio2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C2\":")); jsonMsg.print(CarMon2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C3\":")); jsonMsg.print(CarMon3); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P1\":")); jsonMsg.print(PartMat1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P2\":")); jsonMsg.print(PartMat2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P4\":")); jsonMsg.print(PartMat4); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P10\":")); jsonMsg.print(PartMat10); jsonMsg.println(F("}"));
  jsonOut.clearSpace(jsonMsg.length()); // clear space for this message in the buffer
  jsonOut.print(jsonMsg);
  jsonOut.protect(); // prevent other clearSpace() calls from removing parts of this output
  
  //sending_measurement = false;  // sending data is done
}

The output:

20:23:14.651 ->  2 1 10 9 8 7 6 5 4 3 2 1

20:23:20.533 -> Demo for sending a json measurement string to a Modem.

20:23:20.567 -> Based on the SafeString V2.0.8+ library.

20:23:20.633 -> 

20:23:20.633 -> {"Id":1,"Y":2020,"M":11,"D":12,"Hr":20,"Mi":45,"B1":3.64,"B2":3.67,"T1":20.53,"H1":66.16,"T2":20.92,"H2":56.05,"O1":23.00,"O1g":46.00,"O1t":20.00,"O1h":65.00,"N1":0.00,"N1g":0.00,"N1t":20.00,"N1h":64.00,"C1":6285.00,"C1g":6.00,"C1t":20.00,"C1h":65.00,"O2":8883,"N2":16527,"C2":2331,"C3":22000,"P1":3.85,"P2":4.07,"P4":4.07,"P10":4.07}

20:23:22.624 -> {"Id":1,"Y":2020,"M":11,"D":12,"Hr":20,"Mi":45,"B1":3.64,"B2":3.67,"T1":20.53,"H1":66.16,"T2":20.92,"H2":56.05,"O1":23.00,"O1g":46.00,"O1t":20.00,"O1h":65.00,"N1":0.00,"N1g":0.00,"N1t":20.00,"N1h":64.00,"C1":6285.00,"C1g":6.00,"C1t":20.00,"C1h":65.00,"O2":8883,"N2":16527,"C2":2331,"C3":22000,"P1":3.85,"P2":4.07,"P4":4.07,"P10":4.07}

20:23:24.619 -> {"Id":1,"Y":2020,"M":11,"D":12,"Hr":20,"Mi":45,"B1":3.64,"B2":3.67,"T1":20.53,"H1":66.16,"T2":20.92,"H2":56.05,"O1":23.00,"O1g":46.00,"O1t":20.00,"O1h":65.00,"N1":0.00,"N1g":0.00,"N1t":20.00,"N1h":64.00,"C1":6285.00,"C1g":6.00,"C1t":20.00,"C1h":65.00,"O2":8883,"N2":16527,"C2":2331,"C3":22000,"P1":3.85,"P2":4.07,"P4":4.07,"P10":4.07}

20:23:26.644 -> {"Id":1,"Y":2020,"M":11,"D":12,"Hr":20,"Mi":45,"B1":3.64,"B2":3.67,"T1":20.53,"H1":66.16,"T2":20.92,"H2":56.05,"O1":23.00,"O1g":46.00,"O1t":20.00,"O1h":65.00,"N1":0.00,"N1g":0.00,"N1t":20.00,"N1h":64.00,"C1":6285.00,"C1g":6.00,"C1t":20.00,"C1h":65.00,"O2":8883,"N2":16527,"C2":2331,"C3":22000,"P1":3.85,"P2":4.07,"P4":4.07,"P10":4.07}
24

The message, when I want to use jsonOut.connect(ModemPort) instead:

Arduino: 1.8.13 (Linux), Board: "Arduino Mega or Mega 2560, ATmega2560 (Mega 2560)"

/home/user/Arduino/sketch_dec25a/sketch_dec25a.ino: In function 'void setup()':
sketch_dec25a:92:28: error: no matching function for call to 'BufferedOutput::connect(SoftwareSerial&)'
   jsonOut.connect(ModemPort);
                            ^
In file included from /home/user/Arduino/sketch_dec25a/sketch_dec25a.ino:5:0:
/home/user/Arduino/libraries/SafeString/src/BufferedOutput.h:87:10: note: candidate: void BufferedOutput::connect(HardwareSerial&)
     void connect(HardwareSerial& _serial); // the output to write to, can also read from
          ^~~~~~~
/home/user/Arduino/libraries/SafeString/src/BufferedOutput.h:87:10: note:   no known conversion for argument 1 from 'SoftwareSerial' to 'HardwareSerial&'
/home/user/Arduino/libraries/SafeString/src/BufferedOutput.h:96:10: note: candidate: void BufferedOutput::connect(Stream&, uint32_t)
     void connect(Stream& _stream, const uint32_t baudRate);
          ^~~~~~~
/home/user/Arduino/libraries/SafeString/src/BufferedOutput.h:96:10: note:   candidate expects 2 arguments, 1 provided
exit status 1
no matching function for call to 'BufferedOutput::connect(SoftwareSerial&)'


This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.
25

use

jsonOut.connect(ModemPort,9600); // for Streams that are not HardwareSerial
// Streams don't have the the availableForWrite() method
26

drmpf:
use

jsonOut.connect(ModemPort,9600); // for Streams that are not HardwareSerial

// Streams don't have the the availableForWrite() method

Thanks again for the fast response.
Hmm, when I try this suggestion, it looks like my Mega resets all the time:

02:50:37.665 -> Demo for sending a json measurement string to a Modem.
02:50:37.731 -> B⸮.⸮⸮⸮ 9 8 7 6 5 4 3 2 1
02:50:42.713 -> Demo for sending a json measurement string to a Modem.
02:50:42.779 -> B⸮.⸮⸮⸮ 9 8 7 6 5 4 3 2 1
02:50:47.760 -> Demo for sending a json measurement string to a Modem.
02:50:47.826 -> B⸮.⸮⸮⸮ 9 8 7 6 5 4 3 2 1
02:50:52.840 -> Demo for sending a json measurement string to a Modem.
02:50:52.873 -> B⸮.⸮⸮⸮ 9 8 7 6 5 4 3 2 1
27

Opps Use V2.0.10 of the library that fixes that.
Available now from https://www.forward.com.au/pfod/ArduinoProgramming/SafeString/index.html
and later via Arduino Library manager.

Here is a sample sketch using two BufferedOutputs and a test for missed json output

// comment this line out to use real sensor data
#define TEST_DATA
#include "SafeString.h"  // download from Arduino Library manager
#include "SafeStringStream.h" // included in V2.0.10 of SafeString
#include "BufferedOutput.h" // included in V2.0.10 of SafeString
#include "loopTimer.h" // install loopTimer from zip file
#include "SoftwareSerial.h"   // https://www.arduino.cc/en/Reference/SoftwareSerial communication between Arduino and Module

///  from https://www.forward.com.au/pfod/ArduinoProgramming/RealTimeArduino/index.html

createBufferedOutput(jsonOut, 400, DROP_UNTIL_EMPTY);

createBufferedOutput(output, 400, DROP_UNTIL_EMPTY);

// Softwareserial objects
SoftwareSerial ModemPort(11, 12);    // RX Pin, TX Pin

const int Number = 1;

// Simulated measurement intervall
unsigned long startMillis;
const unsigned long measurementPeriod = 2000;

#ifdef TEST_DATA // replace measurements with test data
float Bat1 = 3.64; float Bat2 = 3.67;

// 3 minute means of measurements
float t1 = 20.53; float h1 = 66.16;
float t2 = 20.92; float h2 = 56.05;

float o31 = 23; float o31_ug = 46;
float o31_t = 20; float o31_h = 65;
float no21 = 0; float no21_ug = 0;
float no21_t = 20; float no21_h = 64;
float co1 = 6285; float co1_mg = 6;
float co1_t = 20; float co1_h = 65;
float o32 = 8883; float no22 = 16527;
float co2 = 2331; float co3 = 22000;
float PM1 = 3.85 ; float PM2 = 4.07;
float PM4 = 4.07; float PM10 = 4.07;

// timestamp in measurement string
int year = 2020;
byte month = 11;
byte day = 12;
byte hour = 20;
byte minute = 45;


#else  // use real inputs from sensors

float Bat1; float Bat2;

// 3 minute means of measurements
float t1; float h1;
float t2; float h2;

float o31; float o31_ug;
float o31_t; float o31_h;
float no21; float no21_ug;
float no21_t; float no21_h;
float co1; float co1_mg;
float co1_t; float co1_h;
float o32; float no22; float co2;
float PM1; float PM2;
float PM4; float PM10;

// timestamp in measurement string
int year;
byte month;
byte day;
byte hour;
byte minute;

#endif


//============

void setup() {
  Serial.begin(9600);
  ModemPort.begin(9600);

  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }

  Serial.println();
  Serial.println(F("Demo for sending a json measurement string to a Modem."));
  Serial.println(F("Based on the SafeString V2.0.10+ library."));
  Serial.println();
  jsonOut.connect(ModemPort, 9600);
  output.connect(Serial);
  SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!

}

//============

void loop() {
  jsonOut.nextByteOut(); // output next buffered chars. Need to call this at least once each loop
  output.nextByteOut();
  loopTimer.check(output);
  if (millis() - startMillis >= measurementPeriod) {
    startMillis = millis();
    SendDataModem(t1, h1, t2, h2, o31, o31_ug, o31_t, o31_h, no21, no21_ug, no21_t, no21_h, co1, co1_mg, co1_t, co1_h, o32, no22, co2, co3, PM1, PM2, PM4, PM10, year, month, day, hour, minute, Bat1, Bat2);
  }
}


//============
void SendDataModem(float t1, float h1, float t2, float h2, float Oz1, float Oz1_ug, float Oz1_temp, float Oz1_hum, float NitDio1, float NitDio1_ug, float NitDio1_temp, float NitDio1_hum, float CarMon1, float CarMon1_mg, float CarMon1_temp, float CarMon1_hum, int16_t Oz2, int16_t NitDio2, int16_t CarMon2, int16_t CarMon3, float PartMat1, float PartMat2, float PartMat4, float PartMat10, int yyyy, byte mon, byte dd, byte hh, byte mm, float AdaBat, float ModemBat) {
  cSF(jsonMsg, 400);
  jsonMsg.print(F("{\"Id\":")); jsonMsg.print(Number); jsonMsg.print(F(","));
  jsonMsg.print(F("\"Y\":")); jsonMsg.print(yyyy); jsonMsg.print(F(","));
  jsonMsg.print(F("\"M\":")); jsonMsg.print(mon); jsonMsg.print(F(","));
  jsonMsg.print(F("\"D\":")); jsonMsg.print(dd); jsonMsg.print(F(","));
  jsonMsg.print(F("\"Hr\":")); jsonMsg.print(hh); jsonMsg.print(F(","));
  jsonMsg.print(F("\"Mi\":")); jsonMsg.print(mm); jsonMsg.print(F(","));
  jsonMsg.print(F("\"B1\":")); jsonMsg.print(AdaBat); jsonMsg.print(F(","));
  jsonMsg.print(F("\"B2\":")); jsonMsg.print(ModemBat); jsonMsg.print(F(","));
  jsonMsg.print(F("\"T1\":")); jsonMsg.print(t1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"H1\":")); jsonMsg.print(h1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"T2\":")); jsonMsg.print(t2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"H2\":")); jsonMsg.print(h2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1\":")); jsonMsg.print(Oz1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1g\":")); jsonMsg.print(Oz1_ug); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1t\":")); jsonMsg.print(Oz1_temp); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O1h\":")); jsonMsg.print(Oz1_hum); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1\":")); jsonMsg.print(NitDio1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1g\":")); jsonMsg.print(NitDio1_ug); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1t\":")); jsonMsg.print(NitDio1_temp); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N1h\":")); jsonMsg.print(NitDio1_hum); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1\":")); jsonMsg.print(CarMon1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1g\":")); jsonMsg.print(CarMon1_mg); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1t\":")); jsonMsg.print(CarMon1_temp); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C1h\":")); jsonMsg.print(CarMon1_hum); jsonMsg.print(F(","));
  jsonMsg.print(F("\"O2\":")); jsonMsg.print(Oz2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"N2\":")); jsonMsg.print(NitDio2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C2\":")); jsonMsg.print(CarMon2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"C3\":")); jsonMsg.print(CarMon3); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P1\":")); jsonMsg.print(PartMat1); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P2\":")); jsonMsg.print(PartMat2); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P4\":")); jsonMsg.print(PartMat4); jsonMsg.print(F(","));
  jsonMsg.print(F("\"P10\":")); jsonMsg.print(PartMat10); jsonMsg.println(F("}"));

  jsonOut.clearSpace(jsonMsg.length()); // clear space for this message in the buffer
  if (jsonOut.availableForWrite() < jsonMsg.length()) {
    output.println("missed output");
  } else {
    output.println("jsonOut");
  }
  jsonOut.print(jsonMsg);
  jsonOut.protect(); // prevent other clearSpace() calls from removing parts of this output

  //sending_measurement = false;  // sending data is done
}

And some sample output

Demo for sending a json measurement string to a Modem.
Based on the SafeString V2.0.10+ library.

jsonOut
jsonOut
jsonOut
loop uS Latency
 5sec max:14300 avg:38
 sofar max:14300 avg:38 max - prt:1692
jsonOut
28

Oh, thank you! You are so quick. :slight_smile: I will play with the new library then. Thank you for the example too.

29

drmpf:
Opps Use V2.0.10 of the library that fixes that.
Available now from The SafeString alternative to Arduino Strings for Beginners Safe, Robust, Debuggable replacement String class for Arduino
and later via Arduino Library manager.

Here is a sample sketch using two BufferedOutputs and a test for missed json output

// comment this line out to use real sensor data

#define TEST_DATA
#include "SafeString.h"  // download from Arduino Library manager
#include "SafeStringStream.h" // included in V2.0.10 of SafeString
#include "BufferedOutput.h" // included in V2.0.10 of SafeString
#include "loopTimer.h" // install loopTimer from zip file
#include "SoftwareSerial.h"   // https://www.arduino.cc/en/Reference/SoftwareSerial communication between Arduino and Module

///  from Simple Multitasking Arduino on any board without using an RTOS

createBufferedOutput(jsonOut, 400, DROP_UNTIL_EMPTY);

createBufferedOutput(output, 400, DROP_UNTIL_EMPTY);

// Softwareserial objects
SoftwareSerial ModemPort(11, 12);    // RX Pin, TX Pin

const int Number = 1;

// Simulated measurement intervall
unsigned long startMillis;
const unsigned long measurementPeriod = 2000;

#ifdef TEST_DATA // replace measurements with test data
float Bat1 = 3.64; float Bat2 = 3.67;

// 3 minute means of measurements
float t1 = 20.53; float h1 = 66.16;
float t2 = 20.92; float h2 = 56.05;

float o31 = 23; float o31_ug = 46;
float o31_t = 20; float o31_h = 65;
float no21 = 0; float no21_ug = 0;
float no21_t = 20; float no21_h = 64;
float co1 = 6285; float co1_mg = 6;
float co1_t = 20; float co1_h = 65;
float o32 = 8883; float no22 = 16527;
float co2 = 2331; float co3 = 22000;
float PM1 = 3.85 ; float PM2 = 4.07;
float PM4 = 4.07; float PM10 = 4.07;

// timestamp in measurement string
int year = 2020;
byte month = 11;
byte day = 12;
byte hour = 20;
byte minute = 45;

#else  // use real inputs from sensors

float Bat1; float Bat2;

// 3 minute means of measurements
float t1; float h1;
float t2; float h2;

float o31; float o31_ug;
float o31_t; float o31_h;
float no21; float no21_ug;
float no21_t; float no21_h;
float co1; float co1_mg;
float co1_t; float co1_h;
float o32; float no22; float co2;
float PM1; float PM2;
float PM4; float PM10;

// timestamp in measurement string
int year;
byte month;
byte day;
byte hour;
byte minute;

#endif

//============

void setup() {
 Serial.begin(9600);
 ModemPort.begin(9600);

for (int i = 10; i > 0; i--) {
   Serial.print(' '); Serial.print(i);
   delay(500);
 }

Serial.println();
 Serial.println(F("Demo for sending a json measurement string to a Modem."));
 Serial.println(F("Based on the SafeString V2.0.10+ library."));
 Serial.println();
 jsonOut.connect(ModemPort, 9600);
 output.connect(Serial);
 SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!

}

//============

void loop() {
 jsonOut.nextByteOut(); // output next buffered chars. Need to call this at least once each loop
 output.nextByteOut();
 loopTimer.check(output);
 if (millis() - startMillis >= measurementPeriod) {
   startMillis = millis();
   SendDataModem(t1, h1, t2, h2, o31, o31_ug, o31_t, o31_h, no21, no21_ug, no21_t, no21_h, co1, co1_mg, co1_t, co1_h, o32, no22, co2, co3, PM1, PM2, PM4, PM10, year, month, day, hour, minute, Bat1, Bat2);
 }
}

//============
void SendDataModem(float t1, float h1, float t2, float h2, float Oz1, float Oz1_ug, float Oz1_temp, float Oz1_hum, float NitDio1, float NitDio1_ug, float NitDio1_temp, float NitDio1_hum, float CarMon1, float CarMon1_mg, float CarMon1_temp, float CarMon1_hum, int16_t Oz2, int16_t NitDio2, int16_t CarMon2, int16_t CarMon3, float PartMat1, float PartMat2, float PartMat4, float PartMat10, int yyyy, byte mon, byte dd, byte hh, byte mm, float AdaBat, float ModemBat) {
 cSF(jsonMsg, 400);
 jsonMsg.print(F("{"Id":")); jsonMsg.print(Number); jsonMsg.print(F(","));
 jsonMsg.print(F(""Y":")); jsonMsg.print(yyyy); jsonMsg.print(F(","));
 jsonMsg.print(F(""M":")); jsonMsg.print(mon); jsonMsg.print(F(","));
 jsonMsg.print(F(""D":")); jsonMsg.print(dd); jsonMsg.print(F(","));
 jsonMsg.print(F(""Hr":")); jsonMsg.print(hh); jsonMsg.print(F(","));
 jsonMsg.print(F(""Mi":")); jsonMsg.print(mm); jsonMsg.print(F(","));
 jsonMsg.print(F(""B1":")); jsonMsg.print(AdaBat); jsonMsg.print(F(","));
 jsonMsg.print(F(""B2":")); jsonMsg.print(ModemBat); jsonMsg.print(F(","));
 jsonMsg.print(F(""T1":")); jsonMsg.print(t1); jsonMsg.print(F(","));
 jsonMsg.print(F(""H1":")); jsonMsg.print(h1); jsonMsg.print(F(","));
 jsonMsg.print(F(""T2":")); jsonMsg.print(t2); jsonMsg.print(F(","));
 jsonMsg.print(F(""H2":")); jsonMsg.print(h2); jsonMsg.print(F(","));
 jsonMsg.print(F(""O1":")); jsonMsg.print(Oz1); jsonMsg.print(F(","));
 jsonMsg.print(F(""O1g":")); jsonMsg.print(Oz1_ug); jsonMsg.print(F(","));
 jsonMsg.print(F(""O1t":")); jsonMsg.print(Oz1_temp); jsonMsg.print(F(","));
 jsonMsg.print(F(""O1h":")); jsonMsg.print(Oz1_hum); jsonMsg.print(F(","));
 jsonMsg.print(F(""N1":")); jsonMsg.print(NitDio1); jsonMsg.print(F(","));
 jsonMsg.print(F(""N1g":")); jsonMsg.print(NitDio1_ug); jsonMsg.print(F(","));
 jsonMsg.print(F(""N1t":")); jsonMsg.print(NitDio1_temp); jsonMsg.print(F(","));
 jsonMsg.print(F(""N1h":")); jsonMsg.print(NitDio1_hum); jsonMsg.print(F(","));
 jsonMsg.print(F(""C1":")); jsonMsg.print(CarMon1); jsonMsg.print(F(","));
 jsonMsg.print(F(""C1g":")); jsonMsg.print(CarMon1_mg); jsonMsg.print(F(","));
 jsonMsg.print(F(""C1t":")); jsonMsg.print(CarMon1_temp); jsonMsg.print(F(","));
 jsonMsg.print(F(""C1h":")); jsonMsg.print(CarMon1_hum); jsonMsg.print(F(","));
 jsonMsg.print(F(""O2":")); jsonMsg.print(Oz2); jsonMsg.print(F(","));
 jsonMsg.print(F(""N2":")); jsonMsg.print(NitDio2); jsonMsg.print(F(","));
 jsonMsg.print(F(""C2":")); jsonMsg.print(CarMon2); jsonMsg.print(F(","));
 jsonMsg.print(F(""C3":")); jsonMsg.print(CarMon3); jsonMsg.print(F(","));
 jsonMsg.print(F(""P1":")); jsonMsg.print(PartMat1); jsonMsg.print(F(","));
 jsonMsg.print(F(""P2":")); jsonMsg.print(PartMat2); jsonMsg.print(F(","));
 jsonMsg.print(F(""P4":")); jsonMsg.print(PartMat4); jsonMsg.print(F(","));
 jsonMsg.print(F(""P10":")); jsonMsg.print(PartMat10); jsonMsg.println(F("}"));

jsonOut.clearSpace(jsonMsg.length()); // clear space for this message in the buffer
 if (jsonOut.availableForWrite() < jsonMsg.length()) {
   output.println("missed output");
 } else {
   output.println("jsonOut");
 }
 jsonOut.print(jsonMsg);
 jsonOut.protect(); // prevent other clearSpace() calls from removing parts of this output

//sending_measurement = false;  // sending data is done
}



And some sample output

Demo for sending a json measurement string to a Modem.
Based on the SafeString V2.0.10+ library.

jsonOut
jsonOut
jsonOut
loop uS Latency
5sec max:14300 avg:38
sofar max:14300 avg:38 max - prt:1692
jsonOut

Thank you! I love how you can play with your different Serials in such a flexible way with the library. I was just thinking about how to check, if the Softwareserial works, because unfortunately I forgot my modem at another place. Using Serial as a debug tool, printing an "ok" message, when the whole json string is parked in jsonOut is a good idea. If that fails, I could just try to park the measurement string again in jsonOut till I can finally send it.
Now only the receive of the timestamp from the modem is left via the software serial. I already began coding that. As soon, as I feel comfortable with my results, I will post my solution. Thanks again!

30

Timesstamp string: Modem > Arduino

Okay, I think I've managed to write a working first version for updating the RTC connected with the Arduino. I get the timestamp from the modem, which is connected to the internet.

I'm thinking about making the whole thing more safe though. In your version beforehand for the gas sensors, you checked for empty fields, right? I know that there is a field for every time element (year, day, hour, minute and so on). "No field should be empty" should be one "gate", before updating the RTC.

Do you have more suggestions, what I should check before updating the RTC? I think you have a better idea about what can go wrong while handling serial data like that.

Thanks for any input!

// comment this line out to use real serial1,2,3 input data
#define TEST_DATA
#include "SafeString.h"  // download from Arduino Library manager
#include "SafeStringStream.h" // included in V2.0.7 of SafeString
#include "BufferedOutput.h" // included in V2.0.7 of SafeString
#include "loopTimer.h" // install loopTimer from zip file
#include "SoftwareSerial.h"   // https://www.arduino.cc/en/Reference/SoftwareSerial communication between Arduino and GPy-Module

///  from https://www.forward.com.au/pfod/ArduinoProgramming/RealTimeArduino/index.html

createBufferedOutput(tsOut, 60, DROP_UNTIL_EMPTY);
char tsdelimiters[] = ">";

// to handle inputs
const size_t maxMsgLength = 32; // length of largest command to be recognized, can handle longer input but will not tokenize it.
createSafeString(tsInput, maxMsgLength + 1); //  to read input cmd, large enough to hold longest cmd + leading and trailing delimiters
createSafeString(tsReceived, maxMsgLength + 1); // for parsing, capacity should be >= input
// set to true to skip input until the first >
bool tsSkipToDelimiter = false; // bool variable to hold the skipToDelimiter state across calls to readUntilToken()


#ifdef TEST_DATA // replace timestamps from serial with test timestamp
char ts[] =  "2010,12,10,12,42,55>";

cSFP(sfModemPortTs, ts); // wrap the data
cSF(tsRxBuf, 64); // simulate Software Serial buffer size
SafeStringStream ModemPort(sfModemPortTs, tsRxBuf); // put it in a SafeStringStream

#else  // use real inputs from other device
// Softwareserial objects
SoftwareSerial ModemPort(11, 12);    // RX Pin, TX Pin

#endif


//============

void setup() {
  Serial.begin(9600);

  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }

  Serial.println();
  Serial.println(F("Demo for receiving a timestamp string from another device."));
  Serial.println(F("Based on the SafeString V2.0.10+ library."));
  Serial.println();
  tsOut.connect(Serial);
  SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!
  ModemPort.begin(9600);
}

//============

void loop() {
  tsOut.nextByteOut(); // output next buffered chars. Need to call this at least once each loop
  //loopTimer.check(tsOut);

    if (tsInput.readUntilToken(ModemPort, tsReceived, tsdelimiters, tsSkipToDelimiter, false)) { // echo on, recycles the input back to the SafeStringStream
      tsOut.print("  Recv. TimeStr : "); tsOut.print(tsReceived);
      updateRTC(" >>>>> Parsed : ", tsReceived);
    }
  

}


// This parser does not change the data
void updateRTC(const char *title, SafeString &timestamp) { // split the data into its parts
  int ModYear, ModMonth, ModDay, ModHour, ModMinute, ModSecond; // temp results

  cSF(sfField, 20); // temp SafeString to received fields
  char delims[] = ",>"; // fields delimited by ,
  bool returnEmptyFields = true; // return empty field for ,,
  size_t idx = 0;
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
  sfField.toInt(ModYear);
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
  sfField.toInt(ModMonth);
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
  sfField.toInt(ModDay);
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
  sfField.toInt(ModHour);
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
  sfField.toInt(ModMinute);
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
  sfField.toInt(ModSecond);
 

  cSF(newTime, 100);
  newTime.print(title);
  newTime.print(F(" Year:")); newTime.print(ModYear);
  newTime.print(F(" Month:")); newTime.print(ModMonth);
  newTime.print(F(" Day:")); newTime.print(ModDay);
  newTime.print(F(" Hour:")); newTime.print(ModHour);
  newTime.print(F(" Minute:")); newTime.print(ModMinute);
  newTime.print(F(" Second:")); newTime.print(ModSecond);

  newTime.println();
  tsOut.clearSpace(newTime.length()); // clear space for this message in the buffer
  tsOut.print(newTime);
  tsOut.protect(); // prevent other clearSpace() calls from removing parts of this output

  //newTimestamp = false;
  //update RTC
  //rtc.adjust(DateTime(ModYear, ModMonth, ModDay, ModHour, ModMinute, ModSecond));
}

The output for the test data:

17:35:27.074 ->  10 10 9 8 7 6 5 4 3 2 1

17:35:32.974 -> Demo for receiving a timestamp string from another device.

17:35:33.041 -> Based on the SafeString V2.0.10+ library.

17:35:33.074 -> 

17:35:33.140 ->   Recv. TimeStr : 2010,12,10,12,42,55 >>>>> Parsed :  Year:2010 Month:12 Day:10 Hour:12 Minute:42 Second:55
31

A few of thoughts.

  1. the terminating > looks artificial. What does your modem actually send? If you have control over that choose a format that is a) easy to parse, b) easy to detect missing data,
    GPS data is terminated by \n and starts with $G... , in the first field. In your case you could terminate the RTC with \n and either add a start field OR easier just check for the first field value >= 2020 and then count and validate the rest of the fields.

  2. I would count the fields to check I was not missing part of a message. I would also do range checks on data e.g. year >=2020, month between 1 and 12, etc

  3. Try testing with a partial first line, i.e. your Arduino start up half way through an RTC message and with invalid data, non-numbers, fields missing, fields out-of-range

  4. How often do the messages arrive?

32

drmpf:
A few of thoughts.

  1. the terminating > looks artificial. What does your modem actually send? If you have control over that choose a format that is a) easy to parse, b) easy to detect missing data,
    Yes, you are actually right. Well spotted. :slight_smile: I'm using another MCU with internet connectivity (Ive called it modem), which sends my data to a server. Basically I save the time elements in the string on that MCU. I defined the endmarker (">") myself . Originally in my older code, I preparded the timestring like that on the other MCU <2020,12,26,14,34,42>, because I've used the Serial Input Basics tutorial as a reference: Serial Input Basics - updated - Introductory Tutorials - Arduino Forum
    That means, I have the full control about the way how the time string looks. What format would you suggest? Could I improve anything here?

drmpf:
GPS data is terminated by \n and starts with $G... , in the first field. In your case you could terminate the RTC with \n and either add a start field OR easier just check for the first field value >= 2020 and then count and validate the rest of the fields. 2) I would count the fields to check I was not missing part of a message. I would also do range checks on data e.g. year >=2020, month between 1 and 12, etc
I've actually thought about validating the timestring in regards of the latter suggestion of you. Do you mean I should do a check like that:
1, >=2020 ist the first check
2. check the amount of fields (in my case 6)
3. I create the sum of the time elements (sum(current timestamp) < sum(received timestamp)) -> Year_1 + month_1 + day_1 + hour_1 + minute_1 + second_1 < Year_2 + month_2 + day_2 + hour_2+ minute_2 + second_2 (The RTC DS3231 has drifting issues so I'm pretty sure, that would work)
4. I check if the elements are in the right range as you suggested

  1. Try testing with a partial first line, i.e. your Arduino start up half way through an RTC message and with invalid data, non-numbers, fields missing, fields out-of-range
    Great suggestion!

  2. How often do the messages arrive?
    I'm planning to update the RTC once a week, because in the datasheet, it was mentioned that the RTC can have a drift between 1 and 2 minutes after a year. Because of the Softwareserial, I have to be careful with the timings. I don't want to receive, while I still try to send data (I want to send data every 3 minutes). Receive and Transmit dont work at the same time with Softwareserial as far as I understood. I will work with flags here.

Thanks for the input! I made my answers to your questions bold.

33

Regarding the check of the number of expected fields: I think that should be the first thing I should do. When I don't have the right amount of fields anyways, I waste time in the loop. Like parsing the first 5 fields, while the 6th isn't even there makes me lose time.

I've thought about implementing that for counting the fields. In the end, I have to set the idx back to Zero, so I can parse timestamp string from the beginning.

  size_t idx = 0;
  // check if the expected fields exist, there should be 6 non-empty fields
  for (int i = 0; i < 6; i++) {
    idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
    if (sfField.isEmpty()) {
      return false;
    }
  }  
  idx = 0; // Index back to Zero for the actual parsing of the timestamp
34

You could sync sending the RTC timestamp back each time you receive a json msg.
Doing that you could remove the RTC module altogether and just use the millis() to adjust the time between RTC updates. The Mega crystal should be accurate to 1sec over 3mins.

Here is another alternative for parsing using nextToken() instead of stoken() and with some more test data. But feel free to continue using stoken()

// comment this line out to use real serial1,2,3 input data
#define TEST_DATA
#include "SafeString.h"  // download from Arduino Library manager
#include "SafeStringStream.h" // included in V2.0.7 of SafeString
#include "BufferedOutput.h" // included in V2.0.7 of SafeString
#include "loopTimer.h" // install loopTimer from zip file
#include "SoftwareSerial.h"   // https://www.arduino.cc/en/Reference/SoftwareSerial communication between Arduino and GPy-Module

///  from https://www.forward.com.au/pfod/ArduinoProgramming/RealTimeArduino/index.html

createBufferedOutput(tsOut, 60, DROP_UNTIL_EMPTY);

// to handle inputs
const size_t maxMsgLength = 32; // length of largest command to be recognized, can handle longer input but will not tokenize it.
createSafeString(tsInput, maxMsgLength + 1); //  to read input cmd, large enough to hold longest cmd + leading and trailing delimiters
createSafeString(tsReceived, maxMsgLength + 1); // for parsing, capacity should be >= input
// set to true to skip input until the first >
bool tsSkipToDelimiter = false; // bool variable to hold the skipToDelimiter state across calls to readUntilToken()

char tsdelimiters[] = " ";
#ifdef TEST_DATA // replace timestamps from serial with test timestamp
char ts[] =  // end with space   OR could be \n if tsdelimiters is \n  OR something else
  "2020,12,10,12,42,55 " // full RTC
  "10,12,10,12,42,56 " // missing begin
  "2020,12,10,12" // missing begin end
  "2020,12,10,12,42,58 " // full RTC but skipped due to missing end of previous record
  "2020,12,10,12,42,59 "; // full RTC but skipped due to missing end of previous record

cSFP(sfModemPortTs, ts); // wrap the data
cSF(tsRxBuf, 64); // simulate Software Serial buffer size
SafeStringStream ModemPort(sfModemPortTs, tsRxBuf); // put it in a SafeStringStream

#else  // use real inputs from other device
// Softwareserial objects
SoftwareSerial ModemPort(11, 12);    // RX Pin, TX Pin

#endif
//============

void setup() {
  Serial.begin(115200); // use highest available for debug output
  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }

  Serial.println();
  Serial.println(F("Demo for receiving a timestamp string from another device."));
  Serial.println(F("Based on the SafeString V2.0.10+ library."));
  Serial.println();
  tsOut.connect(Serial);
  SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!
  ModemPort.begin(9600);
}
//============

void loop() {
  tsOut.nextByteOut(); // output next buffered chars. Need to call this at least once each loop
  //loopTimer.check(tsOut);

  if (tsInput.readUntilToken(ModemPort, tsReceived, tsdelimiters, tsSkipToDelimiter, false)) { // echo on, recycles the input back to the SafeStringStream
    tsOut.print("  Recv. TimeStr : "); tsOut.print(tsReceived);
    if (updateRTC(tsReceived)) {
      tsOut.println(" RTC OK");
      //update RTC
      //rtc.adjust(DateTime(ModYear, ModMonth, ModDay, ModHour, ModMinute, ModSecond));
    } else {
      tsOut.println(" RTC err:");
    }
  }
}

// This parser uses nextToken() which removes data as it processes it
// stoken() uses less cpu cycles and gives more detail but you have to look after the idx variable yourself
bool updateRTC(SafeString &timestamp) { // split the data into its parts
  int ModYear, ModMonth, ModDay, ModHour, ModMinute, ModSecond; // temp results
  
  timestamp += ','; // add a final field terminator so last field will be returned by nextToken
  // stoken returns the last field even when it does not have a terminator so += ',' not needed if using stoken
  
  cSF(sfField, 20); // temp SafeString to received fields
  // note nextToken will not return empty field for ,, but will return sfField for , ,
  // not a problem since we are counting fields and validating ranges

  // sfField will be empty if no more fields and toInt will fail
  // sfField is always cleared at the start of nextToken( ) method
  timestamp.nextToken(sfField, ',');  // returns true if have field, else false. BUT just ignore return and try to convert sfField
  if (!sfField.toInt(ModYear)) { // fails if empty or not an int.
    // error msg here
    return false;
  }
  // range test here
  if ((ModYear < 2020 || ModYear > 2030)) {
    // error msg here
    return false;
  }
  // range test here
  timestamp.nextToken(sfField, ',');
  if (!sfField.toInt(ModMonth)) {
    // error msg here
    return false;
  }
  // range test here
  timestamp.nextToken(sfField, ',');
  if (!sfField.toInt(ModDay)) {
    // error msg here
    return false;
  }
  // range test here
  timestamp.nextToken(sfField, ',');
  if (!sfField.toInt(ModHour)) {
    // error msg here
    return false;
  }
  // range test here
  timestamp.nextToken(sfField, ',');
  if (!sfField.toInt(ModMinute)) {
    // error msg here
    return false;
  }
  // range test here
  timestamp.nextToken(sfField, ',');
  if (!sfField.toInt(ModSecond)) {
    // error msg here
    return false;
  }
  // range test here

  // this is informational so just print what we can and don't worry of some is missing
  tsOut.print(" >>>>> Parsed : ");
  tsOut.print(F(" Year:")); tsOut.print(ModYear);
  tsOut.print(F(" Month:")); tsOut.print(ModMonth);
  tsOut.print(F(" Day:")); tsOut.print(ModDay);
  tsOut.print(F(" Hour:")); tsOut.print(ModHour);
  tsOut.print(F(" Minute:")); tsOut.print(ModMinute);
  tsOut.print(F(" Second:")); tsOut.print(ModSecond);
  tsOut.println();
  return true;
}

the output is

Demo for receiving a timestamp string from another device.
Based on the SafeString V2.0.10+ library.
  Recv. TimeStr : 2020,12,10,12,42,55 >>>>> Parsed :  Year:2020 Month:12 Day:10 Hour:12 Minute:42 Second:55
 RTC OK
  Recv. TimeStr : 10,12,10,12,42,56 RTC err:
  Recv. TimeStr : 2020,12,10,122020,12,10,12,42,58 RTC err:
  Recv. TimeStr : 2020,12,10,12,42,59 >>>>> Parsed :  Year:2020 Month:12 Day:10 Hour:12 Minute:42 Second:59
 RTC OK
35

Re wasting cpu time skipping empty fields, from Donald Knuth Is the Root of All Premature Optimization - Jason Sachs
Premature optimization is the root of all evil. Always remember the three rules of optimization!

  • Don’t optimize.

  • If you are an expert, see rule #1

  • If you are an expert and can justify the need, then use the following procedure:

  • Code it unoptimized

  • determine how fast is “Fast enough”–Note which user requirement/story requires that metric.

  • Write a speed test

  • Test existing code–If it’s fast enough, you’re done.

  • Recode it optimized

  • Test optimized code. IF it doesn’t meet the metric, throw it away and keep the original.

  • If it meets the test, keep the original code in as comments

36

Okay, the first iteration of the sketch based on the advise you gave for updating RTC (The checks to be more precise). I have a weird issue though:

01:42:20.991 ->  10 10 9 8 7 6 5 4 3 2 1

01:42:26.901 -> Demo for receiving a timestamp string from another device.

01:42:26.968 -> Based on the SafeString V2.0.10+ library.

01:42:27.001 -> 

01:42:27.067 -> !!Hour wrong range!!TS bad data :2020,12,10,13,42,55

The sketch has a problem with >=0 (greater than or equal to zero) in the function updateRTC(). When I remove the = in >=, the problem vanishes. I don't get the issue or maybe I'm too tired at the moment to see the issue here

// comment this line out to use real serial1,2,3 input data
#define TEST_DATA
#include "SafeString.h"  // download from Arduino Library manager
#include "SafeStringStream.h" // included in V2.0.7 of SafeString
#include "BufferedOutput.h" // included in V2.0.7 of SafeString
#include "loopTimer.h" // install loopTimer from zip file
#include "SoftwareSerial.h"   // https://www.arduino.cc/en/Reference/SoftwareSerial communication between Arduino and GPy-Module

///  from https://www.forward.com.au/pfod/ArduinoProgramming/RealTimeArduino/index.html

createBufferedOutput(tsOut, 60, DROP_UNTIL_EMPTY);
char tsdelimiters[] = ">";


// to handle inputs
const size_t maxMsgLength = 32; // length of largest command to be recognized, can handle longer input but will not tokenize it.
createSafeString(tsInput, maxMsgLength + 1); //  to read input cmd, large enough to hold longest cmd + leading and trailing delimiters
createSafeString(tsReceived, maxMsgLength + 1); // for parsing, capacity should be >= input
// set to true to skip input until the first >
bool tsSkipToDelimiter = false; // bool variable to hold the skipToDelimiter state across calls to readUntilToken()


#ifdef TEST_DATA // replace timestamps from serial with test timestamp
char ts[] =  "2020,12,10,13,42,55>";

cSFP(sfModemPortTs, ts); // wrap the data
cSF(tsRxBuf, 64); // simulate Software Serial buffer size
SafeStringStream ModemPort(sfModemPortTs, tsRxBuf); // put it in a SafeStringStream

#else  // use real inputs from other device
// Softwareserial objects
SoftwareSerial ModemPort(11, 12);    // RX Pin, TX Pin

#endif


//============

void setup() {
  Serial.begin(9600);

  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }

  Serial.println();
  Serial.println(F("Demo for receiving a timestamp string from another device."));
  Serial.println(F("Based on the SafeString V2.0.10+ library."));
  Serial.println();
  tsOut.connect(Serial);
  SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!
  ModemPort.begin(9600);
}

//============

void loop() {
  tsOut.nextByteOut(); // output next buffered chars. Need to call this at least once each loop
  //loopTimer.check(tsOut);

  if (tsInput.readUntilToken(ModemPort, tsReceived, tsdelimiters, tsSkipToDelimiter, false)) { // echo on, recycles the input back to the SafeStringStream
    if (updateRTC(" >>>>> Parsed : ", tsReceived)) {
    } else {
        tsOut.print("!!TS bad data :"); tsOut.println(tsReceived);
    }
  }
}

// reason for size_t
// https://stackoverflow.com/questions/55604029/should-i-always-use-size-t-when-indexing-arrays

// This parser does not change the data
bool updateRTC(const char *title, SafeString &timestamp) { // split the data into its parts
  int ModYear, ModMonth, ModDay, ModHour, ModMinute, ModSecond; // temp results
  
  cSF(sfField, 20); // temp SafeString to received fields
  char delims[] = ",>"; // fields delimited by ,
  bool returnEmptyFields = true; // return empty field for ,,
  size_t idx = 0;
  // check if the expected fields exist, there should be 6 non-empty fields
  for (int i = 0; i < 6; i++) {
    idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Gas
    if (sfField.isEmpty()) {
      tsOut.print("!!Empty field");
      return false;
    }
  }  
  idx = 0;
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Year
  if (!sfField.toInt(ModYear)) { // str to int possible?
    tsOut.print("!!Year no number");
    return false;
  }
  else if(!sfField.toInt(ModYear) >= 2020) { // in range?
    tsOut.print("!!year wrong range");
    return false; 
  }
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Month
  if (!sfField.toInt(ModMonth)) { // str to int possible?
    tsOut.print("!!Month no number");
    return false;
  }
  else if(!sfField.toInt(ModMonth) >= 1 && !sfField.toInt(ModMonth) <= 12) { // in range?
    tsOut.print("!!Month wrong range");
    return false;
  }
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Day
  if (!sfField.toInt(ModDay)) { // str to int possible?
    tsOut.print("!!Day no number");
    return false;
  }
  else if(!sfField.toInt(ModDay) >= 1 && !sfField.toInt(ModDay) <= 31) { // in range?
    tsOut.print("!!Day wrong range");
    return false;
  }
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Hour
  if (!sfField.toInt(ModHour)) { // str to int possible?
    tsOut.print("!!Hour no number");
    return false;
  }
  else if(!sfField.toInt(ModHour) >= 0 && !sfField.toInt(ModHour) <= 23) { // in range?
    tsOut.print("!!Hour wrong range");
    return false;
  }
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Minute
  if (!sfField.toInt(ModMinute)) { // str to int possible?
    tsOut.print("!!Minute no number");
    return false;
  }
  else if(!sfField.toInt(ModMinute) >= 0 && !sfField.toInt(ModMinute) <= 59) { // in range?
    tsOut.print("!!Minute wrong range");
    return false;
  }
  idx = timestamp.stoken(sfField, idx, delims, returnEmptyFields); // get Second
  if (!sfField.toInt(ModSecond)) {
    tsOut.print("!!Second no number");
    return false;
  }
  else if(!sfField.toInt(ModSecond) >= 0 && !sfField.toInt(ModSecond) <= 59) { // in range?
    tsOut.print("!!Second false Range");
    return false;
  }

  // else all good print reveived timestamp and update the RTC
  
  cSF(newTime, 150);
  newTime.print(F("  Recv. TimeStr : ")); newTime.print(tsReceived);
  newTime.print(title);
  newTime.print(F(" Year:")); newTime.print(ModYear);
  newTime.print(F(" Month:")); newTime.print(ModMonth);
  newTime.print(F(" Day:")); newTime.print(ModDay);
  newTime.print(F(" Hour:")); newTime.print(ModHour);
  newTime.print(F(" Minute:")); newTime.print(ModMinute);
  newTime.print(F(" Second:")); newTime.print(ModSecond);

  newTime.println();
  tsOut.clearSpace(newTime.length()); // clear space for this message in the buffer
  tsOut.print(newTime);
  tsOut.protect(); // prevent other clearSpace() calls from removing parts of this output

  //newTimestamp = false;
  //update RTC
  //rtc.adjust(DateTime(ModYear, ModMonth, ModDay, ModHour, ModMinute, ModSecond));
}

Thank you for the input again! I will read your suggestions tomorrow. It's a bit late here at the moment and I need some sleep.

37

Ah well you have been caught by the 'helpful' C/C++ conversion of bool to int.

if (!sfField.toInt(ModHour))

puts the result in ModHour iff the toInt returns true.
There after you should test that the ModHour int value is in range, i.e.
if ( (ModHour <0) || (ModHour > 23)) { //error
But in your code

!sfField.toInt(ModHour) >= 0

is comparing the bool returned from sfField.toInt( ) to the int value 0 .
also the ! takes precedence over the >=
I always add ( ) to make it clear (to me) what operation gets done first.

38

Okay,

I think the Sketch for receiving a timestamp from another device via Serial to update a RTC is overall fairly complete now. I've implemented multiple checks for missing data/err data:

  • Check for missing fields
  • Check for too many fields
  • Check for empty fields
  • Check for out of range elements
  • Check for non-numbers in fields

The only check, which is missing (in my opinion), is comparing the sum of the time elements of the current UTC timestamp to the received UTC timestamp. Because the DS3231 has drifting issues according to the data sheet, the sum of the current time elements should be smaller than the sum of the received time elements. I can implement it though, when I have access to all my equipment again. I need my DS3231 RTC for that.

I've added also more Test Data for the different cases I check. Thanks again for the help, @drmpf!

I've uploaded the Code, because it's getting to big for posting it here, but here is the Output:

19:16:48.534 -> Demo for receiving a timestamp string from another device. 
19:16:48.600 -> Based on the SafeString V2.0.10+ library. 
19:16:48.633 ->  
19:16:48.766
 ->   Recv. TimeStr : 2020,12,10,12,42,29 >>>>> Parsed
 :  Year:2020 Month:12 Day:10 Hour:12 Minute:42 Second:29 
19:16:48.899 ->  RTC ok 
19:16:48.899 -> !!Year wrong range!!TS bad data :10,12,10,12,42,30 RTC err 
19:16:49.065 -> !!Too many fields!!TS bad data :2020,12,10,122020,12,10,12,42,32 RTC err 
19:16:49.197
 ->   Recv. TimeStr : 2020,12,10,12,42,33 >>>>> Parsed
 :  Year:2020 Month:12 Day:10 Hour:12 Minute:42 Second:33 
19:16:49.297 ->  RTC ok 
19:16:49.330 -> !!Empty or missing field!!TS bad data :2020,10,12,42,34 RTC err 
19:16:49.397 -> !!Empty or missing field!!TS bad data :2020,,10,12,42,35 RTC err 
19:16:49.496 -> !!Year no number!!TS bad data :a,12,10,12,42,36 RTC err 
19:16:49.596 -> !!Month no number!!TS bad data :2020,a,10,12,42,37 RTC err 
19:16:49.696 -> !!Day no number!!TS bad data :2020,12,a,12,42,38 RTC err 
19:16:49.795 -> !!Hour no number!!TS bad data :2020,12,10,a,42,39 RTC err 
19:16:49.895 -> !!Minute no number!!TS bad data :2020,12,10,12,a,40 RTC err 
19:16:49.961 -> !!Second no number!!TS bad data :2020,12,10,12,42,a RTC err 
19:16:50.094 -> !!Too many fields!!TS bad data :2020,12,10,12,42,42,2020 RTC err 
19:16:50.194 -> !!Year wrong range!!TS bad data :2019,12,10,12,42,43 RTC err 
19:16:50.260 -> !!Month wrong range!!TS bad data :2020,-1,10,12,42,44 RTC err 
19:16:50.360 -> !!Month wrong range!!TS bad data :2020,0,10,12,42,45 RTC err 
19:16:50.426 -> !!Month wrong range!!TS bad data :2020,13,10,12,42,46 RTC err 
19:16:50.493 -> !!Day wrong range!!TS bad data :2020,12,0,12,42,47 RTC err 
19:16:50.560 -> !!Day wrong range!!TS bad data :2020,12,32,12,42,48 RTC err 
19:16:50.626 -> !!Hour wrong range!!TS bad data :2020,12,10,-1,42,49 RTC err 
19:16:50.725 -> !!Hour wrong range!!TS bad data :2020,12,10,24,42,50 RTC err 
19:16:50.758 -> !!Minute wrong range!!TS bad data :2020,12,10,12,-1,51 RTC err 
19:16:50.825 -> !!Minute wrong range!!TS bad data :2020,12,10,12,60,52 RTC err 
19:16:50.891 -> !!Second wrong Range!!TS bad data :2020,12,10,12,42,-1 RTC err 
19:16:50.958 -> !!Second wrong Range!!TS bad data :2020,12,10,12,42,60 RTC err

Read_Timestamp_SF_working.ino (8.08 KB)

39

drmpf:
You could sync sending the RTC timestamp back each time you receive a json msg.
Doing that you could remove the RTC module altogether and just use the millis() to adjust the time between RTC updates. The Mega crystal should be accurate to 1sec over 3mins.

Thank you for that idea. The reason I don't do that is, that I'm highly afraid, that the provider of the network for all my needs in regards of IoT can't give me the stability I need for that. They often have to mantain the network and I'm just scared, that it could become so bad at some point, that my data is lagged. While not having the security of a stable network to update the time often enough, I don't want to rely on the Arduino RTC. The external RTC gives me more time and security in that regard, because it's drifting way less compared to the Arduino RTC I think.
I have another question though. I want to disable all the debug information for my measurement system, when I put it outside. For example, I would like to disable all the commands with:

tsOut.print()

because I jus't don't need them anymore then anyways. I just need them before I deploy my measurement device. What would you suggest? Should I work with #ifdef here?
For instance:

#define DEBUG
........
#ifdef DEBUG
  tsOut.print(...);
#endif

Should I also comment the following line in the end:

 //SafeString::setOutput(Serial); // for SafeString Error msg. NOTE: sent to Serial so will block rest of the loop!!

Thanks again for the help!

40

I don't think comparing the new RTC timestamp to the existing one is a good idea.
The drift can make the DS.. run fast as well as slow.
#ifdef DEBUG looks good
I am using a DS3231 in a commercial project I am doing, but I just let it drift. The user can reset the time if they wish and in any case will be doing if for Daylight Savings twice a year. In my case the seconds are not that important