[SOLVED] Arduino Uno communication with external software

Hello ,
I'm trying to collect the output of a sensor RS232 via a software serial on an Arduino 1. The output collected and the voltage on A0 will be send to a PC using standard serial.
Below my code, this is the best I can do searching on internet :slight_smile: :confused:

#include "RTClib.h" //for timestamp
#include <SoftwareSerial.h>

SoftwareSerial portOne(10, 11); //portOne connected to external sensor 
char inByte;
const byte numChars = 52; //length of string send by external sensor
char receivedChars[numChars]; // an array to store the received data

boolean newData = false;

RTC_DS1307 rtc;
void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);

  while (!Serial); // wait for serial port to connect. Needed for native USB

 portOne.begin(9600);
 
  if (! rtc.begin()) {
    //Serial.println("Couldn't find RTC");
    Serial.flush();
    abort();
  }
   rtc.adjust(DateTime(F(__DATE__), F(__TIME__))); //got some problem with RTC I need to initialize it every time btw..
}
  
void loop() {
  // put your main code here, to run repeatedly:
  DateTime now = rtc.now();
  int a=analogRead(A0); //voltage measure

  recvWithEndMarker();  //routine for getting rs232 data from external sensor

  Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
  Serial.print("\t");
  Serial.print(a);
  Serial.print("\t");
  Serial.write(receivedChars);
  Serial.println();
  newData=false;
  //delay(1000);
  
}
void recvWithEndMarker() {
  static byte ndx = 0;
  char endMarker = '\n';
  char rc;
  
  if (portOne.available() > 0) {
    while (portOne.available() > 0 && newData == false) {
      rc = portOne.read();
      
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else {
        receivedChars[ndx] = '\0'; // terminate the string
        ndx = 0;
        newData = true;
      }
    }
  }
}

Using serial monitor everything works almost fine (I've got some error reading the string from sensor).
I want to use an external software (R with Rstudio, using "serial" package) to read the output from Arduino.
I have some problem, first of all I can't collect all the data send from Arduino, viceversa I need a scanrate of 1 second. I cant' control this time with R so I thought to apply a delay on the Serial.print instruction using an if statement with currentmillis() (without using delay), see code:

void loop() {
  // put your main code here, to run repeatedly:
    recvWithEndMarker();
  if(millis() - lastRefreshTime >= REFRESH_INTERVAL)
  {
    lastRefreshTime += REFRESH_INTERVAL;
    getdata();
  }
}

void getdata() {
  DateTime now = rtc.now();
    int a=analogRead(A0);
    Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
    Serial.print("\t");
    Serial.print(a);
    Serial.print("\t");
    Serial.write(receivedChars);
    Serial.println();
    newData=false;
  }

but in this way the routine for reading on portOne doesn't work well.
Is there any way to "freeze" the output for 1 second without altering the reading on portOne?

By the way the external sensor send message each 0.2 second with a 52 chars string, the end of the string is marked with cr/lf.

In attachment an output from serial monitor for both sketches, see in the second one how the sensor data are shifted each loop.
Thank you for any idea

I find this a bit confusing. Your first program uses recvWithEndMarker() to get data but the snippet of the second program does not - it uses a function called getdata() which does not appear in the first program.

Perhaps you can have another go at describing the requirement and the problem?

There should be no difficulty sending a message to the PC every second but you have not said how often new data will be available, or indeed, where it is to come from.

Note, also, that your first program is not using the variable newData to determine when a new message has arrived and should be printed. That might explain why some of your data seem incorrect.

...R

PS ... I assume you mean an Arduino Uno. It is called Uno in every language.

Hi Robin2 thank you,
I will correct the title.

For the second program I omit some part, below the entire code:

#include "RTClib.h"
#include <SoftwareSerial.h>

SoftwareSerial portOne(10, 11);
char inByte;
const byte numChars = 52;
char receivedChars[numChars]; // an array to store the received data

boolean newData = false;
  static const unsigned long REFRESH_INTERVAL = 1000; // ms
  static unsigned long lastRefreshTime = 0;
  
RTC_DS1307 rtc;
void setup() {
  // put your setup code here, to run once:
Serial.begin(57600);
#ifndef ESP8266
  while (!Serial); // wait for serial port to connect. Needed for native USB
#endif
 portOne.begin(9600);
  if (! rtc.begin()) {
    //Serial.println("Couldn't find RTC");
    Serial.flush();
    abort();
  }
   rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
}
  
void loop() {
  // put your main code here, to run repeatedly:
    recvWithEndMarker();
  if(millis() - lastRefreshTime >= REFRESH_INTERVAL)
  {
    lastRefreshTime += REFRESH_INTERVAL;
    getdata();
  }
}

void getdata() {
  DateTime now = rtc.now();
    int a=analogRead(A0);
    Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
    Serial.print("\t");
    Serial.print(a);
    Serial.print("\t");
    Serial.write(receivedChars);
    Serial.println();
    newData=false;
  }

void recvWithEndMarker() {
 static byte ndx = 0;
 char endMarker = '\n';
 char rc;
 
  if (portOne.available() > 0) {
           while (portOne.available() > 0 && newData == false) {
 rc = portOne.read();

 if (rc != endMarker) {
 receivedChars[ndx] = rc;
 ndx++;
 if (ndx >= numChars) {
 ndx = numChars - 1;
 }
 
 }
 else {
 receivedChars[ndx] = '\0'; // terminate the string
 ndx = 0;
 newData = true;
 }
 }
}
}

Triyng to be more exhaustive:
-Arduino Uno with an external sensor RS232 on pins 10,11 via Softwareserial
-voltage measure on Analog pin0
-I don't use the IDE for plotting data, I use Rstudio with a dedicated library called "serial"
-With Rstudio I wanto to log the data form external sensor and analog pin each second
-new data will be available from the external sensor each 0.2-0.3 seconds, see the third column on the first attachment, the field "H2940" is such an internal counter of the sensor. So my intention is to capture one row each second and paste it with the analog read (second column).
others columns came all from the sensor.

In Rstudio i can't set the "delay" so I collect to many data per seconds.

Robin2:
Note, also, that your first program is not using the variable newData to determine when a new message has arrived and should be printed. That might explain why some of your data seem incorrect.

Honestly this code is a copy-paste job :slight_smile: but I supposed that the recvWithEndMarker() will looping until the conclusuin of the sensor string ( with endMarker="\n").

The PC knows what time it is.
There is not need to use a RTC and include that in the information from the sensor.

Now that I can see the whole program I have to ask what exactly this means (from Original Post)

but in this way the routine for reading on portOne doesn't work well.

I know nothing about Rstudio so I must rely on you to describe the problem in as much detail as possible.

By the way, if the sensor is sending data to the recvWithEndMarker() function several times per second the way your program is written it will only capture the first item after the 1 second time has elapsed. Or, put another way, when it comes time to send the next chunk of data to Rstudio the data in receivedChars will not be as fresh as it could be - but maybe that does not matter.

...R

ieee488:
The PC knows what time it is.
There is not need to use a RTC and include that in the information from the sensor.

of course, I used an rtc shield for the future purpose to log data in stand alone mode

Robin2:
By the way, if the sensor is sending data to the recvWithEndMarker() function several times per second the way your program is written it will only capture the first item after the 1 second time has elapsed. Or, put another way, when it comes time to send the next chunk of data to Rstudio the data in receivedChars will not be as fresh as it could be - but maybe that does not matter.

...R

True, but doesn't matter if data is 1 second old...

Rstudio is a similar program of Python, less powerfull but great for statistics and data analysis. I think that Python's library "serial" is the same for Rstudio (with necessary language changes). I've seen some Python's code to manage Arduino communication, they are pretty the same for Rstudio.

If I use the second program I see on serial monitor that the message from sensor is shifted of ncharacters (see the output in the second attachment).

With the firts program all the values are collected almost right (see next in the comment) and with Rstudio I can receive all data, If I use a delay of 1 second in Rstudio many readings are corrupted and, don't know why, it collects 6-7 observation with the same second in time stamp, maybe there is a serial buffer?
Below my code in Rstudio, it's really simple (code's comments after # character):

##from Rstudio## 

#initialise connection with Arduino
con<-serialConnection(port="com18",
                      mode="9600,n,8,1",
                      translation = "cr", #supposed from last Serial.println() in arduino code will be "lf" but doesn't work
                      buffering="none")

open(con) #open the connection
stopTime <- Sys.time() + 30 #30 seconds of fetching
while (Sys.time() < stopTime){
  data <- read.serialConnection(con) #read data from arduino
  
  if(nchar(data)>9){ #work around because some reading are shifted and uncomlpeted
    data<-stringr::str_split(data,pattern = "\t") #split string into substring using tab character as separator
    
    a<-data[[1]][1]           #
    a<-gsub("\n","",a)        #
    a<-gsub("T"," ",a)        #
    # a<-as.POSIXct(a)        # ##data manipulation on received and splitted string
    b<-data[[1]][2]           #
    cc<-data[[1]][3]          #
    d<-data[[1]][3]           #
    data<-as.data.frame(cbind(a,b,cc,d),stringsAsFactors = F) # table generation adding 1 row each iteration
    log <- rbind(log, data)                                   #
    Sys.sleep(1) #wait 1 second before next read
  }
}

(see the output of Rstudio code in attachment).

So I have some question:

  • why Arduino changes some character collected from sensor? (see "H" character in the first and second attachments, sometimes is printed as È)
  • does Arduino need an handshake to communicate with PC (I've seen some tutorial with "rtscts" handshake?
  • why data from sensor are shifted with the second program? It's like Arduino doesn't respect the if statement if (rc != endMarker)

Thanks for your help

Note: In attachment what I see on Rstudio output using the code above:
-First column is date-time from Arduino RTC
-Second column is arduino millis() output at the moment of printing data with Serial.print()
-Third column is arduino millis() rounded in seconds at the moment of printing data
As you cans see even with a delay (Sys.sleep(1) in Rstudio) of 1 second it collect many data with the same time-stamp (value on third column doesn't change)

Image from Reply #6 so we don't have to download it. See this Simple Image Posting Guide

...R

I'm finding all this very confusing - sorry.

What do you see if the Arduino output is displayed on the Arduino Serial Monitor? If that shows the correct data then the problem is with your Rstudio program and I can't help with that.

I note that the millis() value is not incrementing in steps of 1000 - seems to be more like steps of about 80. In your next Reply post the Arduino program that produced the output in the picture in Reply #7

...R

Robin2:
I'm finding all this very confusing - sorry.

What do you see if the Arduino output is displayed on the Arduino Serial Monitor? If that shows the correct data then the problem is with your Rstudio program and I can't help with that.

I note that the millis() value is not incrementing in steps of 1000 - seems to be more like steps of about 80. In your next Reply post the Arduino program that produced the output in the picture in Reply #7

...R

The code is the same of the first attachment (with new serial.print for millis evaluate), so in Arduino there isn't any kind of delay, this coul be the reason for the increment of 80.
I manage the delay in Rstudio with sys.sleep. I agree with you that the problem isn't with Arduino, but I can't understand where Rstudio get all the data with the same time-stamp. I suppose that Arduino send in continuous fresh data (and that's true I've checked the output with PuTTy) for some reasons there is like a data buffer. More confusionig is that I have tryed Rstudio code with a simple rs232 converter and I'm able to fecth fresh data each second..

This i s the code:

#include "RTClib.h" //for timestamp
#include <SoftwareSerial.h>

SoftwareSerial portOne(10, 11); //portOne connected to external sensor 
char inByte;
const byte numChars = 45; //length of string send by external sensor
char receivedChars[numChars]; // an array to store the received data
//String receivedChars; // an array to store the received data
boolean newData = false;

RTC_DS1307 rtc;
void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);

  while (!Serial); // wait for serial port to connect. Needed for native USB

 portOne.begin(9600);
 
  if (! rtc.begin()) {
    //Serial.println("Couldn't find RTC");
    
    Serial.flush();
    abort();
  }
  
   rtc.adjust(DateTime(F(__DATE__), F(__TIME__))); //got some problem with RTC I need to initialize it every time btw..
}
  
void loop() {
  // put your main code here, to run repeatedly:
  DateTime now = rtc.now();
  int a=analogRead(A0); //voltage measure

  recvWithEndMarker();  //routine for getting rs232 data from external sensor
  Serial.flush();
  Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
  Serial.print("\t");
  //Serial.print(a);
  Serial.print(millis());
   Serial.print("\t");
  Serial.print(millis()/1000);
  Serial.print("\t");
  Serial.write(receivedChars);
  Serial.println("");
  newData=false;
  //delay(1000);
  
}
void recvWithEndMarker() {
  static byte ndx = 0;
  char endMarker = '\n';
  char rc;

  
  if (portOne.available() > 0) {
    while (portOne.available() > 0 && newData == false) {
      rc = portOne.read();
      
      if (rc != endMarker) {
        
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
     
      else {
      receivedChars[ndx] = '\0'; // terminate the string
      ndx = 0;
      newData = true;
      }
    }
  }
}

I'm triyng a work-around with native Arduino code for sending data to pc each second with this code:

#include "RTClib.h"
#include <SoftwareSerial.h>

SoftwareSerial portOne(10, 11);
char inByte;
const byte numChars = 45;
char receivedChars[numChars]; // an array to store the received data

boolean newData = false;
static const unsigned long REFRESH_INTERVAL = 1000; // ms
static unsigned long lastRefreshTime = 0;

RTC_DS1307 rtc;
void setup() {
  // put your setup code here, to run once:
  Serial.begin(57600);
  #ifndef ESP8266
  while (!Serial); // wait for serial port to connect. Needed for native USB
  #endif
  portOne.begin(9600);
  if (! rtc.begin()) {
   //Serial.println("Couldn't find RTC");
    Serial.flush();
  abort();
  }
  rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
}

void loop() {
  // put your main code here, to run repeatedly:
  recvWithEndMarker();
  if(millis() - lastRefreshTime >= REFRESH_INTERVAL){
    lastRefreshTime += REFRESH_INTERVAL;
    getdata();
    }
  }
  
void getdata() {
  DateTime now = rtc.now();
  int a=analogRead(A0);
  Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
  Serial.print("\t");
  Serial.print(a);
  Serial.print("\t");
  Serial.write(receivedChars);
  Serial.println();
  newData=false;
}
  
void recvWithEndMarker() {
 static byte ndx = 0;
char endMarker = '\n';
char rc;

if (portOne.available() > 0) {
  while (portOne.available() > 0 && newData == false) {
    rc = portOne.read();
    
    if (rc != endMarker) {
      receivedChars[ndx] = rc;
      ndx++;
      if (ndx >= numChars) {
        ndx = numChars - 1;
      }
    }
    else {
      receivedChars[ndx] = '\0'; // terminate the string
      ndx = 0;
      newData = true;
    }
  }
  } 
}

but in this case I have problems with the reading of sensor data, the output string from portOne is uncorrect:
-this is the correct pattern
H1387 0000 0967 0970 +31 0000 02 01 002 0000
-this is the output on serial monitor
H1388 0000 0967 0H1396 0000 0967 0970 +31 00

The behaviour in serial monitor is like this:
correct string
uncorrect string
correct string
uncorrect string
.....and so on

this is the output

EDIT: I think that Arduino doesn't recognize the start of the communication with the sensor, I'm trying to start the fetching only if the first char is "H"..

Please slow down a bit. It's difficult to keep up when you post multiple Replies without allowing time for a response.

The code is the same of the first attachment (with new serial.print for millis evaluate), so in Arduino there isn't any kind of delay, this coul be the reason for the increment of 80.

I thought you had moved on from there to use the program with an interval determined by millis() as in your Reply #2. There seems little to be gained by sticking with the code that does not do what you want.

...R

I'm trying to solve the problem both working on Arduino side and Rstudio side. With first attempt I try to manage the output rate from arduino with the if statement before the print() commands<-I've got the problem with sensor data fetching.
With the second attempt I try to solve the problem using a delay between readings in Rstudio<- I've got the problem that many readings are inconsistent.

So..let'se if there is a solution for the first attempt. In this case I can't understand why the recvWithEndMarker() loop could be influenced by the if statement on he main loop

paolo_cristoforetti:
With the second attempt I try to solve the problem using a delay between readings in Rstudio

That approach is a complete waste of time. The listening device should be listening all the time.

Write an Arduino program that sends a message once per second and see what that looks like in Rstudio compared to the output in the image in Reply #7

...R

PS ... Please note that the image in Reply #10 is unreadable. Don't post images of text. Just copy and paste the text.

Robin2:
That approach is a complete waste of time. The listening device should be listening all the time.

Write an Arduino program that sends a message once per second and see what that looks like in Rstudio compared to the output in the image in Reply #7

...R

PS ... Please note that the image in Reply #10 is unreadable. Don't post images of text. Just copy and paste the text.

OK,learned the lesson.
This is an output with 1 sec intervall
2020-07-08T21:41:02 1023 9001 9 H2161 0000 0937 0H2165 0000 0937 0942 +32 00
2020-07-08T21:41:03 1023 10001 10 H2170 0000 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:04 1023 11001 11 H2171 0000 0937 0H2175 0000 0937 0942 +32 00
2020-07-08T21:41:05 1023 12001 12 H2180 0000 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:06 1023 13001 13 H2181 0000 0937 0H2185 0000 0937 0942 +32 00
2020-07-08T21:41:07 1023 14001 14 H2190 0000 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:08 1023 15001 15 H2191 0000 0937 0H2195 0000 0937 0942 +32 00
2020-07-08T21:41:09 1023 16001 16 H2200 0000 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:10 1023 17014 17 H2201 0000 0937 0205 0000 0937 0942 +32 0000
2020-07-08T21:41:11 1022 18024 18 H20 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:12 1023 19021 19 H2211 0000 0937 0942 +3200 02 01 002 0000
2020-07-08T21:41:13 1023 20022 20 H2216 0000 0937 094200
2020-07-08T21:41:14 1023 21018 21 H2221 0000 0937 0942 +32 0000 02 01 002H2226
2020-07-08T21:41:15 1023 22011 22 H2232 0000 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:16 1023 23004 23 H2233 0000 0937 0H2236 0000 0937 0942 +32 00
2020-07-08T21:41:17 1023 24001 24 H2241 0000 0937 0942 +32 0000 02 01 002 0000
2020-07-08T21:41:18 1023 25001 25 H2242 0000 0937 0H2246 0000 0937 0942 +32 00

1st column time stamp
2nd column analogread0
3rd column millis()
4th millis()/1000
5th sensor read id
6th not important
7th & 8th sensor measumerment
As you can see sensor data are completely inconsistent, this is a correct row:
2020-07-08T21:41:03 1023 10001 10 H2170 0000 0937 0942 +32 0000 02 01 002 000

paolo_cristoforetti:
As you can see sensor data are completely inconsistent, this is a correct row:
2020-07-08T21:41:03 1023 10001 10 H2170 0000 0937 0942 +32 0000 02 01 002 000

You seem to be making progress at last.

What values do you see if you send the output to the Arduino Serial Monitor ?

ALSO - please post the Arduino program that produced the output in Reply #14

...R

That's the code:

#include "RTClib.h"
#include <SoftwareSerial.h>

SoftwareSerial portOne(10, 11);
char inByte;
const byte numChars = 45;
char receivedChars[numChars]; // an array to store the received data

boolean newData = false;
static const unsigned long REFRESH_INTERVAL = 1000; // ms
static unsigned long lastRefreshTime = 0;

RTC_DS1307 rtc;
void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  #ifndef ESP8266
  while (!Serial); // wait for serial port to connect. Needed for native USB
  #endif
  portOne.begin(9600);
  if (! rtc.begin()) {
   Serial.println("Couldn't find RTC");
    Serial.flush();
  abort();
  }
  rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
}
void recvWithEndMarker() {
 static byte ndx = 0;
char endMarker = '\n';
char rc;

if (portOne.available() > 0) {
  
  while (portOne.available() > 0 && newData == false) {
    rc = portOne.read();
    if (rc != endMarker) {
      receivedChars[ndx] = rc;
      ndx++;
      if (ndx >= numChars) {
        ndx = numChars - 1;
      }
    }
    else {
      receivedChars[ndx] = '\0'; // terminate the string
      ndx = 0;
      newData = true;
    }
  }
  } 
}
void loop() {
  // put your main code here, to run repeatedly:
  recvWithEndMarker();
  if(millis() - lastRefreshTime >= REFRESH_INTERVAL){
    lastRefreshTime += REFRESH_INTERVAL;
    getdata();
    }
  }
  
void getdata() {
  DateTime now = rtc.now();
  a=analogRead(A0);
  Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
  Serial.print("\t");
  Serial.print(a);
  Serial.print("\t");
  Serial.write(receivedChars);
  Serial.println();
  newData=false;
}

What values do you see if you send the output to the Arduino Serial Monitor ?

I get the same output of Reply#14, with the same alternating pattern

It's not surprising that it is not working because you are still not checking for newData == true. I mentioned this in Reply #1

...R

I put the check for new data in the getdata void

void getdata() {
  DateTime now = rtc.now();
  a=analogRead(A0);
  if(newData==true){
  Serial.print(now.timestamp(DateTime::TIMESTAMP_FULL));
  Serial.print("\t");
  Serial.print(a);
  Serial.print("\t");
  Serial.write(receivedChars);
  Serial.println();
  newData=false;
  }

nothing changed, the output is still weird with alternating rows.
But for whatI understand from the code the check for new data is unusless because the program will remain in the recvWithEndMarker() until it received the terminating char from serial

void recvWithEndMarker() {
static byte ndx = 0;
char endMarker = '\n';
char rc;

if (portOne.available() > 0) {

while (portOne.available() > 0 && newData == false) {
rc = portOne.read();
if (rc != endMarker) {
receivedChars[ndx] = rc;
ndx++;
if (ndx >= numChars) {
ndx = numChars - 1;
}
}
else {
receivedChars[ndx] = '\0'; // terminate the string
ndx = 0;
newData = true;
}
}
}
}

paolo_cristoforetti:
But for whatI understand from the code the check for new data is unusless because the program will remain in the recvWithEndMarker() until it received the terminating char from serial

The code only remains in recvWithEndMarker() until the Serial Input Buffer has been emptied. That can happen even though the complete message has not been received because the Arduino works very much faster than serial data arrives. It may take several calls to recvWithEndMarker() before all of the message has arrived.

The purpose of the newData variable is to enable the main code to know when the message is complete and when it is safe to read the content of receivedChars[].

And if a second message starts arriving before the program has changed newData back to false the second message will be ignored by recvWithEndMarker() so as to allow time for the first message to be processed.

Please post your complete program and not just a snippet.

...R