filtering serial data

hello

i am looking to "filter" out some information from a serial read, since i i dont need the start/stop bits and the checksum bit

part of the code i am currently using

if (RS485Serial.available())
  {
     while(RS485Serial.available() && getdata!='d')
     {
        getdata=RS485Serial.read(); 
        digitalWrite(Pin13LED, HIGH); 
        Serial.print(getdata);        
        digitalWrite(Pin13LED, LOW); 
     }
    Serial.println(" "); 
  }

and i get a responce like this

00.1 022 +24.4 5E*48

but i would very much like to split it up so i only get

00.1 022 +24.4

could someone help me out with what command would be best suited for this.
the serial data string is a constant length, and format (BBVV.V DDD SSS.S CCBB)

The strtok() function can be used to split a string.

groundFungus:
The strtok() function can be used to split a string.

that would split it up in to nice sections, but how would i get rid of the startbit ? there is nothing separating it from the first reading.

00.1

it might be easier to store each section separately, is there a command that will store the first letter in 1 variable and then the next 4 in variable 2 and so forth ?

then it would be simple to jsut serial.print the variables i need, and forget about the rest.

What is the 'startbit' that you are referring to? BB? Slightly strange that BB is also the end of the message, but OK.

May we assume that getdata is just a byte? Not posting full code makes it difficult to judge. If it's indeed a single byte, you first want to collect a complete message before you apply strtok(). Although the examples in Serial Input Basics might not fully apply because BB also seems to be the end of the message, it will give you an idea how to save to an array.

Once you have an array, you can simply point to the 3rd element to skip the starting BB. Below code also removes the BB at the end

char message[32] = "BBVV.V DDD SSS.S CCBB";

void setup()
{
  Serial.begin(57600);

  // strip BB from the end of the message
  message[strlen(message) - 2] = '\0';
  
  // not needed; just for demo
  // data points to the 3rd element of the received message
  char *data = &message[2];
  Serial.println(data);
  
  // parse the received message; either use &message[2] or use data variable
  char *token;
  token = strtok(&message[2], " ");
  while (token != NULL)
  {
    Serial.println(token);
    token = strtok(NULL, " ");
  }
}

void loop()
{
}

Output

VV.V DDD SSS.S CC
VV.V
DDD
SSS.S
CC

okay some extra background info :slight_smile:

i have a sonic wind sensor that i am getting data from using an RS485 connection. so i have no way of changing the sent data at the source.

the data sent by the sensor, is sent after it receives a command to do so (10TR2). the response is then sent in the following format (taken from the manual)

(STX)VV.V DDD TTT.T SS*CC(CR)(ETX)
STX = startbit (serial monitor shows it as a square)
V = Windspeed
D = wind direction
T = temperature
S = status byte
C = Check sum (EXOR link)
EXT = endbit (serial monitor shows it as a square)

as mentioned before, i just need the 3 values V,D and T.

coming from the sensor it looks like this

00.1 022 +24.4 5E*48

and here is the complete code to help clarify.

#include <SoftwareSerial.h>

#define SSerialRX        10  
#define SSerialTX        11  
#define SSerialTxControl 3   
#define RS485Transmit    HIGH
#define RS485Receive     LOW
#define Pin13LED         13

SoftwareSerial RS485Serial(SSerialRX, SSerialTX);

void setup()
{
  Serial.begin(9600);
  pinMode(Pin13LED, OUTPUT);
  pinMode(SSerialTxControl, OUTPUT);
  digitalWrite(SSerialTxControl, RS485Receive);
  RS485Serial.begin(9600);
}

void loop()
{
  char getdata='m';
  digitalWrite(Pin13LED, HIGH);
  digitalWrite(SSerialTxControl, RS485Transmit); 
  RS485Serial.println("10TR2");
  delay(7);
  digitalWrite(Pin13LED, LOW);
  digitalWrite(SSerialTxControl, RS485Receive);     
  
  if (RS485Serial.available())
  {
     while(RS485Serial.available() && getdata!='d')
     {
        getdata=RS485Serial.read(); 
        digitalWrite(Pin13LED, HIGH); 
        Serial.print(getdata);        
        digitalWrite(Pin13LED, LOW); 
     }
    Serial.println(" "); 
  }
  delay(980);
}

but by the sound of it i will need to get it into an array, so i will start reading up on arrays :slight_smile:

The serial input basics code receives the input data into a character array (string) even if the data is sent by a series of prints. The character array is then easy to parse with strtok(). See example #5.
The start and end markers help to make reception robust. If you have control of how the data is sent, then changing the sending code to include start and end markers will make reception more reliable.

If you have a STX and an ETX, you can modify the example 3 in the link that I provided; STX is the start marker and ETX is the end marker; obviously you need to modify the example to use the SoftwareSerial instead of Serial. Example 3 will throw the STX and ETX away so you only have the real message.

For your information, STX and ETX are NOT bits but bytes / characters (8 bits) :wink:

You can keep the while-loop that I showed, but you will need to keep track of which token represents what. Or you can do a few individual calls to strtok. I've done the latter in below code. It uses a function to extract the relevant parameters.

char message[32] = "VV.V DDD TTT.T SS*CC\r";

char velocity[5];
char direction[4];
char temperature[6];

void setup()
{
  Serial.begin(57600);

  if (parseMessage(message) == false)
  {
    // hang forever
    for (;;);
  }

  Serial.print("Velocity =    "); Serial.println(velocity);
  Serial.print("Direction =   "); Serial.println(direction);
  Serial.print("Temperature = "); Serial.println(temperature);

}

void loop()
{
}

/*
  parse received data
  In:
    data
  Returns:
    true on success, else false
  Note:
    fills global variables velocity, direction and temerature
*/
bool parseMessage(char *data)
{
  char *token;
  Serial.print("[Debug]Data: "); Serial.println(data);

  // velocity
  token = strtok(data, " ");
  if (token == NULL)
  {
    Serial.println("Corrupt message; no velocity");
    return false;
  }
  if (strlen(token) > sizeof(velocity) - 1)
  {
    Serial.println("Corrupt message; too many characters for velocity");
    return false;
  }
  strcpy(velocity, token);

  // direction
  token = strtok(NULL, " ");
  if (token == NULL)
  {
    Serial.println("Corrupt message; no direction");
    return false;
  }
  if (strlen(token) > sizeof(direction) - 1)
  {
    Serial.println("Corrupt message; too many characters for direction");
    return false;
  }
  strcpy(direction, token);

  // temperature
  token = strtok(NULL, " ");
  if (token == NULL)
  {
    Serial.println("Corrupt message; no temperature");
    return false;
  }
  if (strlen(token) > sizeof(temperature) - 1)
  {
    Serial.println("Corrupt message; too many characters for temperature");
    return false;
  }
  strcpy(temperature, token);

  return true;
}

thanks i will play around with it and see if i can get it working :slight_smile:

just wondering, the STX and ETX show up as a square in the serial monitor, how can you get the arduino to use that for anything ? if it doesnt have an ASCII sign ?

Those are non-printable characters end serial monitor will indicate that with e.g. the squares.

If you make the changes to example3 to handle STX and ETX instead of < and >, you will see that the Arduino code (that you upload) does something useful with them; that is, it uses them to determine the start and end of the message.

but in the example 3 what do i define as STX and ETX, because i cant seem to get that one working, it seems like it is waiting for the data.

but i can see that the data is transmitted to the arduino (RS485 to USB converter on the same bus)

STX, ETX, NAK etc are well defined terms and have standardised values when dealing with serial comms.

STX is 0x02, ETX is 0x03

Do a search for STX/ETX protocol

apparently just doing copy/paste of the square from the serial monitor works too :slight_smile:

but i will have a look at the protocol to make the code look better :slight_smile:

and then it works just fine removing the the STX and ETX marks

@darrob gave you the hex numbers; use those.

e.g. something like

//char startMarker = '<';
char startMarker = 0x02; // STX

I've commented out the original so you have a reference.

Your instrument provides a serial protocol by framing the data between STX ( start of text ) and ETX ( end of text ) which are values 2 and 3 respectively.

The Arduino has a serial method called readBytesUntil which can handle this kind of protocol very well.

A major plus of this method is that the STX and ETX are discarded and what remains is a zero terminated string.

Example

byte getData[26];
const char *pointer;
const byte STX=2;
const byte ETX=3;

void setup() {
 Serial.begin(9600);
 Serial.setTimeout(10);

}

void loop() {
 if (Serial.available() > 0) {
 
         memset(getData,0,sizeof(getData));
          if(Serial.read()==STX){
          Serial.readBytesUntil(ETX,getData,26);

  pointer = getData;
    while (*pointer) {
          Serial.print(*pointer);
          pointer++; }
          Serial.println(" ");
 }         
 }          
 }

A difference between the methods in the serial input basics tutorial and readBytesUntil() is that readBytesUntil blocks while reading serial and the serial input basics methods do not. It may be the difference between a responsive program and one the seems to lag. Serial input is slow in terms of the Arduino processor. With the serial input basics methods the Arduino can do thousands of things instead of being tied up waiting for serial input.

the response time is not so important in this case, since the plan is to have an update from the sensor every second, and send it on to a website (eventually).

but i see your point, especially if it is part of a bigger system.

i might add in more sensors when i get this working well, (on a website) and basically build it into a complete weather station, but that should still be possible to get all sensor readings and send them on to a python script every second

but i think i might use the readbytesuntil to get rid of the checksum value (last 5 chars before ETX) since they are not realy used for anything in this case.

but i really appreciate the help guys :slight_smile:

@frichness I figured the application was not too time sensitive, good luck with your app.

@groundFungus
All methods block while reading serial unless its a multi threaded system. But I do understand the point you are trying to make.

I have read Robins Serial Basics and I think its a must read, and as Robin noted the different possible scenarios of reading and writing serial are almost endless.

That is why I don't like to dismiss readBytesUntil out of hand. The code for readBytesUntil is easy to read and understand.With a fixed length data packet, as we have in this situation, testing the packet length before a read can make this method as quick if not quicker than other methods.

With readBytesUntil the data we have here (21 bytes) can be transferred from buffer to array under ~0.1mS or if we do it without the packet length test ~20.0mS

I am not saying any particular way is superior to the other but to look at all the options and in this application readBytesUntil may well be suitable for the OP or it may not

sumguy:
All methods block while reading serial unless its a multi threaded system. But I do understand the point you are trying to make.

There's a huge difference between "blocking" while copying data from a dedicated serial buffer to an array controlled by your code and blocking while you're waiting for data to come in through the serial port. The former takes microseconds to copy data that's already in the buffer. The latter takes an indeterminate amount of time.

The code for readBytesUntil is easy to read and understand.

Then it should be an easy matter to convert it to a polled / non-blocking variant.[/quote]

i was looking at this strtok() aswell, and i realy like the idea of splitting the data string up and placing it in an array, it would make it neater when i eventually add more sensors, so that i could just add each sensor reading to the same array. then at the end of the loop, put all the relevant readings into 1 serial print.

but i cant seem to wrap my head around this function at the moment, i am still quite new with this programming, the arduino code in general i can read my way out of, and it sorta makes sense. serial.read - reads what is on the serial port. but this strtok doesnt make much sense, i assume it's in C or something like that. maybe when i get a bit more experience in programming it makes more sense.

this project did sound fairly simple at the beginning, but it would seem that i have underestimated the complexity of the matter, and starting out with a fairly advanced sensor and including RS485, arduino, raspberry and a web server in my first project, might be a bit of a large project to start with. but i guess i have to challenge myself in order to learn :slight_smile: but 1 step at a time.

frichsness:
but i cant seem to wrap my head around this function at the moment, i am still quite new with this programming, the arduino code in general i can read my way out of, and it sorta makes sense. serial.read - reads what is on the serial port. but this strtok doesnt make much sense, i assume it's in C or something like that. maybe when i get a bit more experience in programming it makes more sense.

I gave you two examples for the use of strtok(); the one in reply #6 is tailored to your needs and stores the three parameters in separate variables.

If you don't understand that example, ask. If you don't understand how strtok() works internally, it's a bit outside the scope of the thread but somebody might be willing to explain.