Go Down

Topic: Serial read performance issue - or is it? (Read 178 times) previous topic - next topic

jvoveris

Sep 24, 2020, 03:45 pm Last Edit: Sep 24, 2020, 03:47 pm by jvoveris
Hi All,

I'm trying to decode some messages sent over serial bus from a commercial data logger. Data format here:

https://www.race-technology.com/wiki/index.php/General/SerialDataFormat

There have been attempts to do the same in the past: https://github.com/colinbyrne105/RTArduino

I've failed to get Colin's code to work, and couldn't quite follow his logic, so i gave it a go myself and ended up with this (quite a bit of code that did make sense to me is replicated from Colin's, the rest is written according to my 'way of thinking'):

Code: [Select]

#include <Arduino.h>

#define MaxLength  42 //max message length in bytes
#define MaxMsg    107 //max message id

int buffer[MaxLength] = {0}; //buffer for msg data
unsigned int bytesInBuffer = 0; //buffer size tracker
byte messageLengths[]={9,11,0,7,21,6,6,6,5,14,10,3,0,5,5,5,5,5,0,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,0,11,6,10,10,10,11,11,11,11,11,3,5,30,11,4,4,42,42,3,5,5,5,6,24,3,6,4,4,5,5,5,5,10,5,5,5,5,6,5,4,5,6,5,10,8,0,0,0,19,0,17,9,11,0,0};
int currentMessageLength = 0;    //Nr of bytes in the current message
int bytesNeededInBuffer;         //Nr of bytes needed in buffer to complete message
byte checkSum;         //checksum calculated from bytes in message

bool newData = false;
bool newMsg = false;

unsigned long timeStamp = 0;


bool debug = 0; //eanble debug (serial prints)

//------------------------------------------
// FUNCTIONS
//------------------------------------------

//Function to shift bytes down in message buffer
void moveBytesDown(void){
 for (int i=0; i<=bytesInBuffer; i++){
 buffer[i] = buffer[i+1];
 }//end of for loop to move bytes down
 bytesInBuffer=bytesInBuffer-1;
 newData = false;
 if(debug == 1){
 Serial.print("buffer shifted: ");
 for (int i=0; i<=10; i++){
 Serial.println(buffer[i]);
 }
 }

}


void showNewMsg() {
    if (newMsg == true) {
        Serial.print("got msg: ");
        Serial.println(timeStamp);
        newMsg = false;
    }
}

/*
TIME STAMP DATA
Channel Number 09
Total Length 5 bytes
Channel Data1 Data2 Data3 Checksum
*/
//Decode func - CH09
float decodeTimeStamp(int msgArray[]) {
 float result = 0;
 byte checksum = 0;
 for (int k=0; k<4; k++) {
 checksum += msgArray[k];
 }
 if (checksum == msgArray[4]) {
 result = (msgArray[1] * pow(2,16) + msgArray[2] * pow(2,8) + msgArray[3]);
 return result;
 } else { // invalid checksum
 return -1;
 }
}



//------------------------------------
//End of FUNCTIONS
//------------------------------------


void setup() {
 Serial.begin(115200);
 Serial1.begin(115200);
 Serial.println("Serial ready");
 Serial1.println("Serial 1 ready");
}


void loop() {

 //SERIAL BUFFER FULL CHECK
 if (Serial1.available()>60){      //Check if serial buffer is full
 while(Serial1.available()>0)
 Serial1.read();    //if full, clear buffer to ensure full messages are recorded
 newData = false;
 if(debug == 1){
 Serial.println("buffer full, cleared");
 }
 }

 //SERIAL BUFFER EMPTY CHECK
 if(bytesInBuffer == 0){ //is buffer counter empty?
 if(Serial1.available()>=1){ //is serial data available?
 buffer[0] = Serial1.read(); //read first byte into msg buffer
 bytesInBuffer = 1; //increment buffer counter
 if(debug == 1){
 Serial.print("was empty, buffer: ");
 Serial.println(buffer[0]);
 }
 } //end of serial buffer check
 } //end of buffer counter check

 //MSG ID RANGE CHECK
 if(buffer[0] > MaxMsg){ //is 1st byte outside msg id range?
 moveBytesDown(); //if 'yes', move bytes down in buffer counter
 if(debug == 1){
 Serial.println("outside msg id, move bytes");
 }
 }

 //GET MSG LENGTH
 currentMessageLength = messageLengths[(buffer[0]-1)];      //get message length from msg table
 if(debug == 1){
 if (newData == true){
 Serial.print("required msg length: ");
 Serial.println(currentMessageLength);
 }
 }
 //BUILD MESSAGE
 if (currentMessageLength > bytesInBuffer){     //check if we have enough bytes in our buffer
 bytesNeededInBuffer = currentMessageLength-bytesInBuffer; //if not enough, how many needed
 if (Serial1.available()>=bytesNeededInBuffer){ //check if there are enough bytes in serial array
 int bytesInBufferOld = bytesInBuffer;
 for(int i=bytesInBufferOld; i < currentMessageLength; i++){    //for loop to add bytes to end of our array
 buffer[i]= Serial1.read();
 bytesInBuffer=bytesInBuffer + 1;
 newData = true;
 }//end for loop
 }
 }
 //DO CHEKSUM
 if(newData == true){
 checkSum = 0;      //reset checksum

 for (int i=0; i<(currentMessageLength-1); i++){      //for loop to sum checksum value
 checkSum = checkSum + buffer[i];
 }//end if checksum for loop

 if (checkSum == buffer[currentMessageLength-1]){ //if message has been proven
 if(debug == 1){
 Serial.print("checksum: ");
 Serial.println(checkSum);
 }

 //DO MESSAGE DECODING HERE
 if (buffer[0]==9) {                          //Do we want to use this msg? example msg address 9
 timeStamp = decodeTimeStamp(buffer);
 if(debug == 1){
 Serial.print("timeStamp: ");
 Serial.println(timeStamp);
 }
 newMsg = true;
 }

 //CLEAR MSG BUFFER
 for(int i = 0; i < currentMessageLength; i++){ //clear msg bytes
 moveBytesDown();
 }
 }else{ //if checksum not ok, shift bytes in buffer
 moveBytesDown();
 }
 }

 newData = false;

 showNewMsg();

}


    

I use Eclipse, hence this is not quite 'standard' Arduino code.

The script works, but i have performance issue. Serial1 (this is where data comes in) keeps on overflowing - after reset i get around 10 - 20 messages decoded, then buffer fills up and i don't get any more message decodes. If reset, behaviour repeats itself. This is behaviour with only Serial.print's in showNewMsg() activated, all other serial prints are disabled, and only single message address being decoded.

So im trying to understand whether this is HW limitation (i dont have enough uC HW knowledge to be able to work it out), or my code is so inefficient and takes too long to decode message. This is run on atmega 2560.

Thanks in advance
Jaras

TheMemberFormerlyKnownAsAWOL

Don't use pow() to raise integers to integer powers - you may not get the results you expect.
Please don't PM technical questions - post them on the forum, then everyone benefits/suffers equally

Robin2

I use Eclipse, hence this is not quite 'standard' Arduino code.

The script works, but i have performance issue. Serial1 (this is where data comes in) keeps on overflowing - after reset i get around 10 - 20 messages decoded, then buffer fills up and i don't get any more message decodes.
Is it the input buffer or the output buffer that is filling up?

If the input buffer is filling up then the likely cause is that your program is not emptying it fast enough.

If the output buffer is filling up then it seems likely that the data is coming in more quickly than it can go out and maybe a faster output baud rate would help.


Separately this seems like something that could be done with a few lines of regular Arduino code.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

jvoveris

Don't use pow() to raise integers to integer powers - you may not get the results you expect.
Thanks for advice, will look in to it. I have removed this function completely for now to see if it has an effect on my serial performance, but i still get the issue with buffer overrun.

I have simplified the code as much as i can (i think :D). It now is simply incrementing an int every time one of the valid messages is received (address 09), and prints that int.

Code: [Select]

#include <Arduino.h>
//#include "decodeFuncs.h"

#define MaxLength  42 //max message length in bytes
#define MaxMsg    107 //max message id

int buffer[MaxLength] = {0}; //buffer for msg data
unsigned int bytesInBuffer = 0; //buffer size tracker
byte messageLengths[]={9,11,0,7,21,6,6,6,5,14,10,3,0,5,5,5,5,5,0,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,0,11,6,10,10,10,11,11,11,11,11,3,5,30,11,4,4,42,42,3,5,5,5,6,24,3,6,4,4,5,5,5,5,10,5,5,5,5,6,5,4,5,6,5,10,8,0,0,0,19,0,17,9,11,0,0};
int currentMessageLength = 0;    //Nr of bytes in the current message
int bytesNeededInBuffer;         //Nr of bytes needed in buffer to complete message
byte checkSum;          //checksum calculated from bytes in message

bool newData = false;
bool newMsg = false;

//unsigned long timeStamp = 0;
int result = 0;
int resultOld = 0;


bool debug = 1;
//------------------------------------------
// FUNCTIONS
//------------------------------------------

//Function to shift bytes down in message buffer
void moveBytesDown(void){
for (int i=0; i <= bytesInBuffer; i++){
buffer[i] = buffer[i+1];
}//end of for loop to move bytes down
bytesInBuffer = bytesInBuffer-1;
newData = false;
}

void showNewMsg() {
    if (newMsg == true) {
        Serial.print("got msg: ");
        Serial.println(result);
        newMsg = false;
        resultOld = result;
    }
}



void setup() {
Serial.begin(115200);
Serial1.begin(115200);
Serial.println("Serial ready");
Serial1.println("Serial 1 ready");
}


void loop() {

//SERIAL BUFFER FULL CHECK
if (Serial.available()>60){      //Check if serial buffer is full
while(Serial.available()>0)
Serial.read();    //if full, clear buffer to ensure full messages are recorded
newData = false;
if(debug == 1){
Serial.println("serial out full");
}
}
if (Serial1.available()>60){      //Check if serial buffer is full
while(Serial1.available()>0)
Serial1.read();    //if full, clear buffer to ensure full messages are recorded
newData = false;
if(debug == 1){
Serial.println("serial in full");
}
}

//SERIAL BUFFER EMPTY CHECK
if(bytesInBuffer == 0){ //is buffer counter empty?
if(Serial1.available()>=1){ //is serial data available?
buffer[0] = Serial1.read(); //read first byte into msg buffer
bytesInBuffer = 1; //increment buffer counter
} //end of serial buffer check
} //end of buffer counter check

//MSG ID RANGE CHECK
if(buffer[0] > MaxMsg){ //is 1st byte outside msg id range?
moveBytesDown(); //if 'yes', move bytes down in buffer counter
}

//GET MSG LENGTH
currentMessageLength = messageLengths[(buffer[0]-1)];      //get message length from msg table

//BUILD MESSAGE
if (currentMessageLength > bytesInBuffer){    //check if we have enough bytes in our buffer
bytesNeededInBuffer = currentMessageLength-bytesInBuffer; //if not enough, how many needed
if (Serial1.available()>=bytesNeededInBuffer){ //check if there are enough bytes in serial array
int bytesInBufferOld = bytesInBuffer;
for(int i=bytesInBufferOld; i < currentMessageLength; i++){    //for loop to add bytes to end of our array
buffer[i]= Serial1.read();
bytesInBuffer=bytesInBuffer + 1;
newData = true;
}//end for loop
}
}
//DO CHEKSUM
if(newData == true){
checkSum = 0;      //reset checksum

for (int i=0; i<(currentMessageLength-1); i++){      //for loop to sum checksum value
checkSum = checkSum + buffer[i];
}//end if checksum for loop

if (checkSum == buffer[currentMessageLength-1]){ //if message has been proven

//DO MESSAGE DECODING HERE
if (buffer[0]==9) {                          //Do we want to use this msg? example msg address 9
// timeStamp = decodeTimeStamp(buffer);
result = resultOld+1;
newMsg = true; //got new message
}

//CLEAR MSG BUFFER
for(int i = 0; i < currentMessageLength; i++){ //clear msg bytes
moveBytesDown();
}
}else{ //if checksum not ok, shift bytes in buffer
moveBytesDown();
}
}

newData = false;

if (newMsg == true){
showNewMsg();
}

}




I still get serial input buffer overrun after around 100 - 200 successful messages (takes a few seconds).
I have tried to set output serial to 250000, but still get input buffer overrun on serial1 - which i guess confirms that my reading in is too slow.

Robin - i did go through your serial introduction thread, which was very useful. Unfortunately this particular way of data stream is not covered - here logger is sending a constant byte stream with no start or stop markers, and various messages are not equal in length. Baud is fixed at 115200.
My understanding is (and what my code does i think) to decode messages i need to compare incoming byte against possible messages id's (1 to 107). When incoming byte matches one of id's, i look up required number of bytes to complete that particular msg, and wait until i receive those bytes. Then i calc checksum, and if that confirms, i have a message. Then message is decoded, and message buffer is emptied. Then i start looking for byte matching some msg id again.
I know logger is working as it should - i can monitor serial on PC on their own software including raw serial data, and it all makes sense.

So i guess now im trying to understand whether my code (which i think i now simplified to bare minimum) is somehow very inefficient and blocking receive, or mega2560 is too slow for this task. I feel it is former, but i'm out of ideas on how to speed it up...     
 

Robin2

Robin - i did go through your serial introduction thread, which was very useful. Unfortunately this particular way of data stream is not covered - here logger is sending a constant byte stream with no start or stop markers, and various messages are not equal in length. Baud is fixed at 115200.
There must be some structure to the messages. Can you post a link to the documentation for the data logger?

What is longest message?

Perhaps you could create a circular buffer with (say) 20% more space than the longest message. Then when X bytes are in the Serial Input Buffer copy them to the tail of the circular buffer. While waiting for the next X bytes to arrive you will have time to traverse the buffer looking for the message ID byte.

Another thought is that maybe there is a time gap between the end of one message and the start of the next. If so that could be used to identify the different messages.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

jvoveris

Serial data format here:
https://www.race-technology.com/wiki/index.php/General/SerialDataFormat
My understanding is that there is no time delay or any other marker to help split messages in the data stream, unless im overlooking something - which is quite possible... 

Longest message (id 52) is 67 bytes, but i dont need it. The ones i need are in the 4 - 42 bytes range.

byte messageLengths[]={...} is a message length lookup array in my code. Current code is only decoding  message 9, which is 5 bytes long (it is time stamp sent by logger periodically).

So currently my message buffer is int buffer[42] = {0} (42 bytes is the longest message i'm interested in). And currently code is building a message byte by byte.

Re. circular buffer - could you share a simple example on how this can be implemented?



Robin2

Serial data format here:
What a crap system :)



Quote
Re. circular buffer - could you share a simple example on how this can be implemented?
I don't have an example on hand. The idea is that there is a byte array to represent the buffer and two other variables that hold the position in the buffer of the start and the end of the data. Suppose the buffer is 96 bytes long (you must allow for the long message even if you don't need it).

Suppose that we are in the middle of operations and the bufferHead variable has the value 32 and the bufferTail variable has the value 31. That means that the latest 96 bytes of data data starts at position 32, goes all the way to the end of the array and wraps around to the start and continues to position 31.

Your program then scans the array starting at position 32 and working along looking for a header value. If it finds one it continues on to where it expects the checksum and tries to verify the data. If it verifies it then it copies that data to another place and updates the bufferHead value to the position of the byte after the checksum. If it does not verify then it moves the bufferHead on by one byte and tries to find another header value.

Along with this the program should be checking the Serial Input buffer and copying any bytes that arrive starting at the position after the bufferTail (assuming the bufferHead has moved forward to make some space). Then the bufferTail value is moved along to the position of the most recent byte that was copied over.

I'm assuming in all of this that the search for messages in the circular buffer will happen fast enough to make plenty of space available for the new bytes that arrive in the Serial Input Buffer. I think that's a reasonable assumption.

Hope this makes sense.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

jvoveris


jvoveris

#8
Sep 29, 2020, 01:23 am Last Edit: Sep 29, 2020, 01:27 am by jvoveris
So i didn't quite succeed in implementing the circular buffer - i have implemented the buffer itself quite easily, but managed to confuse myself to death while trying to build the message from that buffer :D Anyway, not all is lost - i have found a major issue in my original code while trying to implement the circ. buffer. Once potential message ID byte was found in original program, it was trying to build a complete message within the single main loop iteration. I have corrected that now, and i get 3-4 steady bytes  in serial in buffer during the stream. So all good for now, but i may try to crack that circular buffer in the future!

Thanks for help!   

Go Up