SDISerial (SDI-12) and HardwareSerial - Lost of data on HardwareSerial

I'm developing an interface based on an Arduino Mega 2560 to convert RS232 data coming from a sensor to SDI-12 that can be read by a data-recorder. The sensor delivers data as a serial ascii stream (50 bytes at 9600 baud every 10 seconds...).

I found the library SDISerial on GitHub - EnviroDIY/Arduino-SDI-12: An Arduino library for SDI-12 communication with a wide variety of environmental sensors. This library provides a general software solution, without requiring any additional hardware.. I've made some additions because the library was built to read SDI-12 informations from a sensor and not to deliver SDI-12 measurements to a data recorder... The class uses the Pin PWM 2 of the Arduino as suggested to be used...

For the connection with the RS232 sensor, I use Serial1 hardware serial interface with max232.

Everything is working but... if I increase the frequency of RS232 data receiving and data-recorder queries, in some cases, I lose some bytes in Serial1 buffer.

I don't think it is because the Arduino is too busy or input Serial1 buffer is full (Buffer size increased to 256 bytes). I'm using two processes that are not blocking and continuously checking SDISerial and Serial1 buffer availability.

Is it possible that there is a conflict between SDISerial class (SDIClass is derived from SoftwareSerial) that use software interrupt and Serial1 when they are concurrently exchanging data.

I thanks you for your suggestions/ideas...

J.F.

Is it possible that there is a conflict between SDISerial class (SDIClass is derived from SoftwareSerial) that use software interrupt and Serial1 when they are concurrently exchanging data.

Since you are modifying the class anyway, you should change the base class. Make it derive from Stream, which is the base class for both SoftwareSerial and HardwareSerial.

Without links to the libraries you are using, and your code, we'd just be guessing what the problem was.

The SDI12 library I use can be found in SDISerial/SDISerial.cpp at master · joranbeasley/SDISerial · GitHub

I just added two methods :

  • sdi_write (identical with sdi_cmd from library but without code to send "Break" not relevant in case of response
  • checkSlave (method I execute in ino main loop to check SDI buffer and process if needed

The code I added to the SDISerial class is

void SDISerial::sdi_write(const char* bytes) {
//clone of sdi_cmd(const char* bytes) but "Break" has been removed because not relevant when responding to a data-recorder query
//detach interrupt for this ...
flush();
detachInterrupt(pin2int[_receivePin]);
char buffer[255];
int i = 0;
char* p = (char*)bytes;
buffer[0] = '\0';
uint8_t oldSREG = SREG;
pinMode(_receivePin, OUTPUT);
while (*p != '\0') {
write(*p++);
}
pinMode(_receivePin, INPUT);

setRX(_receivePin);//, INPUT);
//reattach interrupt so we will see response stream
attachInterrupt(pin2int[_receivePin], handle_interrupt, CHANGE);
}
void SDISerial::initStatusLedPin()
{
if (_statusLedPin != 0)
{
pinMode(_statusLedPin, OUTPUT);
digitalWrite(_statusLedPin, LOW);
}

}

bool SDISerial::checkSlave() //parse SDI buffer and process buffer if SDI command is recognized (M!, D0!, D1!,...)
{
static char acc[20];
static int iChar = 0;
static int sentValue = -1;
char c;
char rep[50];
char buf[10];
static char chr_13[2]={(char)13,'\0'};
static char chr_10[2]={(char)10,'\0'};
bool returnValue = false;
bool responseTooLong = false;
while (available() > 0)
{
c = read()&0x7F; //read one byte in input buffer
if (c != '\0')
{
#if DEBUG == 1
Serial.print(".");
Serial.print(c);
Serial.print("(");
Serial.print((int)c);
Serial.print(")");
#endif
acc[++iChar] = c; //accumulate byte in acc buffer
}
}
if (acc[iChar]=='!' && acc[iChar-1] == 'M') // Measurement request (M!)
{
iChar = -1;
flush();
returnValue = true;
Serial.print("Received M");
rep[0] = '\0';
if (_statusLedPin != 0)
{
digitalWrite(_statusLedPin, HIGH); //switch on SDI-12 info led
}
strncpy(rep, "\0", 1);
sprintf(buf, "%i", _sdiAddress);
strcat(rep, buf);
strcat(rep, "001");
sprintf(buf, "%i", _nbValues);
strcat(rep, buf);
strcat(rep, chr_13);
strcat(rep, chr_10);
#if DEBUG == 1
Serial.print("->SDI:");
Serial.println(rep);
#endif
sdi_write(rep);
sentValue = 0;
}
else
{
if (acc[iChar] == '!' && acc[iChar-2]=='D' ) // Data requests (D0! , D1!,...)
{

iChar = -1;
flush();
rep[0] = '\0';
sprintf(buf, "%i", _sdiAddress);
strcat(rep, buf);
responseTooLong = false;
while (responseTooLong == false && sentValue < _nbValues)
{
sprintf(buf, "%+i", (int)dV[sentValue]);
if ((strlen(rep) + strlen(buf)) < 32) //check if length of data to send is greater that 32 (SDI12 limitation)
{
strcat(rep, buf);
sentValue++;
}
else //Length of data exceed 32... Next data will be sent in next data request
{
responseTooLong = false;
}
}
strcat(rep, chr_13);
strcat(rep, chr_10);
#if DEBUG == 1
Serial.print("->SDI:");
Serial.println(rep);
#endif
sdi_write(rep);
if (_statusLedPin != 0)
{
digitalWrite(_statusLedPin, LOW); //switch off SDI-12 info led
}
}
}
return returnValue;
}

The Hardware serial1 port (_oSerial in code) is checked from ino main loop by method check()

bool SerialListenerClass::check()
{
bool bRet = false;
if (_opened == true)
{
while (_oSerial->available() > 0 && bRet == false) {
char c = _oSerial->read();
if (c == '\n') {
_serialBuf[_serialBufPos++] = '\0';
_serialBufPos = 0;
decodeBuf(); //if \n received, decode input buffer
bRet = true;
}
else
{

_serialBuf[_serialBufPos++] = c;
_serialBuf[_serialBufPos] = '\0';
}

if (_serialBufPos > SERIAL_BUFFER_LENGTH)
_serialBufPos = 0;
}
}
return bRet;
}

void SerialListenerClass::decodeBuf() //parse buffer and extract values that must be kept
{
char acc[20];
int posAcc = 0;
int nbKept = -1;
int nbInBuf = 0;
long v[20];
char toKeep[50]={1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};
char *ptr = _serialBuf;

while (*ptr != '\0')
{
if (*ptr != 32)
{
acc[posAcc++] = *ptr;
}
else
{
if (posAcc > 0)
{
acc[posAcc] = '\0';
if (toKeep[nbInBuf] == 1)
{
nbKept++;

sscanf(acc,"%li",&v[nbKept]);

}
nbInBuf++;
}
posAcc = 0;
}
*ptr++;
}
if (posAcc > 0)
{
if (toKeep[nbInBuf] == 1)
{
acc[posAcc] = '\0';
nbKept++;

sscanf(acc,"%li",&v[nbKept]);

}
nbInBuf++;
}
if (nbInBuf != _nbBufValues)
{
Serial.println("************oupsbuffer length not ok");
_nbNotOk++;
}
else
{
if (nbKept != (_nbValues - 1))
{
Serial.println("**********oupsnb kept values not ok");
_nbNotOk++;
}
else
{
Serial.print("--> AVG:");
_nbOk++;
if (_resetNextData == true)
{
_nbOk = 1;
_nbNotOk = 0;
}
for (int i = 0; i < _nbValues; i++)
{
if (_resetNextData == true)
{
_avgS
.reset();

  • }*
    avgS.addValue((double)v*);
    }
    }_
    resetNextData = false;*
    * }
    }[/td]
    [/tr]
    [/table]
    The ino main loop is
    void loop()
    {
    if (oSDI.checkSlave()== true) //if Data request received from data recorder, update values in SDISerial class to be sent to data recorder*
    * {_
    for (int i = 0 ; i < SERIAL_NB_VALUES;i++)
    _ {
    oSDI.setValueSlave(i,oSerial1.AverageValues(i));
    }
    }
    if (oSerial1.check()== true) //check if data received in Serial1 buffer*
    * {
    }
    }*_
  char buffer[255];
   int i = 0;

What are these unused variables for?

        else
         {

            _serialBuf[_serialBufPos++] = c;
            _serialBuf[_serialBufPos] = '\0';
         }

Regardless of whether there is room, or not. Hmmm...

I don't know how you posted that code, but it was not in code tags.

   char buffer[255];
   int i = 0;

Was existing in the SDISerial class in sdi_cmd(...) that i cloned to build sdi_write(...) from download. It could/should be removed... I think...

while (_oSerial->available() > 0 && bRet == false) {
 char c = _oSerial->read();
 if (c == '\n') {
 _serialBuf[_serialBufPos++] = '\0';
 _serialBufPos = 0;
 decodeBuf(); //if \n received, decode input buffer
 bRet = true;
 }
 else
 {
#if DEBUG == 1
 Serial.print(c);
#endif
 _serialBuf[_serialBufPos++] = c;
 _serialBuf[_serialBufPos] = '\0';
 }

 if (_serialBufPos > SERIAL_BUFFER_LENGTH)
 _serialBufPos = 0;
 }

_SerialBuf length is checked next line... But I'm sure the buffer does not exceed capacity (255) when the problem occurs. Small buffer of 50 char at 9600 baud every 2 seconds... It can't be full. It is not the problem.

Sorry not to use "code tags" in my previous reply. It should be better now...Thanks for your remark.

_SerialBuf length is checked next line.

AFTER you have stepped on memory you don't own.

It can't be full. It is not the problem.

Well, OK.

You need to plan for what CAN happen, though, not what WILL happen.

Thanks for your comment...

Yes, you are right ... the buffer overflow test should be done before adding new character

while (_oSerial->available() > 0 && bRet == false) {
 if (_serialBufPos > SERIAL_BUFFER_LENGTH - 2)
{
#if DEBUG == 1
   Serial.print("************* buffer full !**************");
#endif
   _serialBufPos = 0;
}

 char c = _oSerial->read();
 if (c == '\n') {
 _serialBuf[_serialBufPos++] = '\0';
 _serialBufPos = 0;
 decodeBuf(); //if \n received, decode input buffer
 bRet = true;
 }
 else
 {
#if DEBUG == 1
 Serial.print(c);
#endif
 _serialBuf[_serialBufPos++] = c;
 _serialBuf[_serialBufPos] = '\0';
 }

 }

I will test this change but...

 if (_serialBufPos > SERIAL_BUFFER_LENGTH - 2)

That should be -1, not -2. You only plan to add one character. You plan to move the NULL.

Hello JF,

You should have a look at this post

http://forum.arduino.cc/index.php?topic=37847.0

I understand that SerialReceiveData can be lost if serial data is "arriving" while another interrupt is running.

Can somebody confirm this?

idea: RTS/CTS handshaking ?

Regards

stef

Thank you Stef,

By reading the post you suggested, it seems that my sketch could be in this case...

But I don't see any solution for my application...
I have no choice on selecting higher baud rate and no RS232 handshake is available, ... :confused:

I must use the RS232 sensor as it is and have to deal with that...

Thanks for your investigations.

Regards

J.F.

I'm building a new sketch by replacing the Arduino Mega Serial1 with one I2C UART SC16IS750 chip. (From SparkFun but unfortunatly retired)

I hope it can be THE solution...

J.F.