Processing/Extraction of Serial data

[Bryanpl]

Hello:

I have a GPSDO (GPS disciplined OXCO) that has a RS232 port that can be queried for data, setting etc. I have a Arduino sketch polling the GPSDO every couple of seconds to return data. the eventual goal is to extract the data I need and display it on a LCD.

if (mySerial.available())  //Look for data from GPSDO
   { // Show activity
    byteSend = mySerial.read();    // Read received byte
    Serial.write(byteSend);        // Show on Serial Monitor
    
   }

I am fairly new to this and wondering what the best approach would be to extract specific data out of the data stream. Attached is a jpg of a screen shot that shows the ascii codes i.e. cr/lf In the example below what would be the best way of extracting the Frequency correction value. Should I read the serial data character by character, and say throw out the data before the 10th line feed and read between the 10th and 11th line feed, or should I retrieve the data as a string and somehow extract the data I need from the string. Just not sure what the best way to approach this.

[giovanniguerra]

If I were you I'd put the data you receive byte by byte into a String and then use the SubString function...

String substring Function | Arduino Documentation

...to find out where "freq cor" starts.

Is the text sent always the same size?

I think the main problem would be to know when the string starts, presumably with the ---------- followd by the date...?

[system]

wondering what the best approach would be to extract specific data out of the data stream.

Use the TinyGPS++ library. It comes with examples.

[Bryanpl]

PaulS:
Use the TinyGPS++ library. It comes with examples.

Unfortunately this GPSDO does not seam to support NMEA, or it is NMEA "like". Have tried the TinyGPS++ but it will not communicate.

[Bryanpl]

giovanniguerra:
If I were you I'd put the data you receive byte by byte into a String and then use the SubString function...

https://www.arduino.cc/en/Tutorial/StringSubstring

...to find out where "freq cor" starts.

Is the text sent always the same size?

I think the main problem would be to know when the string starts, presumably with the ---------- followd by the date...?

Thanks, this is a good starting point. the text size is always the same, just the values will change. Suppose I could search for "freq cor = " and then retrieve the data until the crlf .

[dev_1]

Yea, don't use String. You'll eventually be sorry.

You can always do readBytesUntil to get to a particular point, then parse whatever's next. That function just takes a "C string".

But... what device is it, and what kind of protocol does it use? Do you have sample data? If it is emitting NMEA sentences, NeoGPS can read the most common ones. It can also read variants that TinyGPS and TinyGPS++ cannot read.

Cheers,
/dev

[giovanniguerra]

/dev:
Yea, don't use String. You'll eventually be sorry.

Why not use String?

[Bryanpl]

/dev:
Yea, don't use String. You'll eventually be sorry.

You can always do readBytesUntil to get to a particular point, then parse whatever's next. That function just takes a "C string".

But... what device is it, and what kind of protocol does it use? Do you have sample data? If it is emitting NMEA sentences, NeoGPS can read the most common ones. It can also read variants that TinyGPS and TinyGPS++ cannot read.

Cheers,
/dev

The device is a Symmetricom GPSDO /Furuno receiver. Communication is done by a RS232 header, but it is not NMEA standard at the RS232 level, apparently there is possibly some way of hacking the receiver to read NMEA, but just not there yet. Sample data is what I posted in the first message in the attachment.

[Robin2]

Strings (capital S) can cause corruption in the small memory of an Arduino.

You may get some useful ideas in Serial Input Basics

...R

[giovanniguerra]

Robin2:
Strings (capital S) can cause corruption in the small memory of an Arduino.

I was thinking in general, not for serial port processing. Is String then "officially" discouraged? char arrays are safer because static? Is that the reasoning?

[Robin2]

giovanniguerra:
char arrays are safer because static? Is that the reasoning?

Yes

...R

[zoomkat]

I have a GPSDO (GPS disciplined OXCO) that has a RS232 port that can be queried for data, setting etc.

Are you using a converter to convert the rs232 to the arduino TTL format?

[dev_1]

Why not use String?...

This is not a novice topic, which is unfortunate, because String is so easy to use. I've gathered a good sampling of the rationale into one post here. There's lots of reading to do, and the problems described can be hard to visualize. This is an excellent summary of the "life cycle" of a project that starts out using String. Read it and weep.

I was thinking in general, not for serial port processing.

In general, it is strongly discouraged (or even mocked) by the embedded software industry and in applications where high reliability is important, like servers. It is specifically prohibited in certain life-critical systems. But because the Arduino is more of a hobbyist environment, the consequences of a failure are more like, "Oh well, that was interesting" or "Why doesn't it work any more?"

Is String then "officially" discouraged?

Well, if you can identify the official, you can ask them... But if you're asking if the Arduino gods discourage it, the answer is no. String is provided as part of the distribution, and there are Reference pages describing the class. That page is linked to from the lower-case "string" page:

string Description::
Text strings can be represented in two ways. you can use the String data type, which is part of the core as of version 0019, or you can make a string out of an array of type char and null-terminate it. This page described the latter method. For more details on the String object, which gives you more functionality at the cost of more memory, see the String object page.

The "more functionality" part is pure malarky! It also omits that fact that they are indeterminately slower to use. The learning curve for C strings (aka char arrays) can be steeper than String for beginners, and so they may start out with String. As described in the life-cycle link above, it rarely ends well.

char arrays are safer because static? Is that the reasoning?

If by "static", you mean they don't use dynamic memory, the answer is "Yes." Embedded systems must be deterministic to guarantee they will run forever. Dynamic memory is not deterministic -- its state depends on the complete history of String operations, which is usually affected by an input stream.

So if you're just tinkering or doing a simple homework assignment, then go ahead and use String. Nobody's life will be endangered, and you won't let the magic smoke out of the Arduino. But your time would be better spent on C strings, and your program can evolve without mysteriously failing after you add the nth feature and let it run for 17 hours.

Cheers,
/dev

[zoomkat]

giovanniguerra:
I was thinking in general, not for serial port processing. Is String then "officially" discouraged? char arrays are safer because static? Is that the reasoning?

A fairly recent discussion on the subject.

https://forum.arduino.cc/index.php?topic=353834.0

[dev_1]

Sample data is what I posted in the first message in the attachment.

Sorry, but many people (including myself) will not download an image just to look at it once. You can embed it in the post so it is visible to anyone reading the thread:

(Technique described here.)

That looks like a console interface. It doesn't really show the NMEA-like data. I think you can use this interface, but you'd have to program the Arduino to (1) watch for the command prompt with readBytesUntil, (2) send a command, (3) watch for particular fields to come back, again with readBytesUntil.

For non-blocking solutions (i.e., not readBytesUntil), Robin2 has already pointed you to Serial Input Basics. Finite state machines could help, too. Take a look at the Useful Links page.

What model number do you have? If it's the GN or GV 86/87 series, they will emit standard NMEA sentences. NeoGPS is unique in being able to parse the talker IDs, unlike TinyGPS, TinyGPS++ or Adafruit_GPS.

The Furuno devices can also send and receive Proprietary sentences. NeoGPS provides a method to send those configuration sentences from FLASH, saving a bunch of RAM:

     gps.send_P( &mySerial, F("PERDCFG,NMEAOUT,GST,0") ); // Disable GST sentence

To parse the proprietary sentences, you can derive classes as shown in a NeoGPS example, PUBX. The Furuno devuces have a lot of capabilities, so I'm not sure if you want to do a lot or just a little. If you just need one thing, it might be quicker to use the very-lean readBytesUntil, followed by a few parseInt calls.

However, if you want to verify the checksums, get several fields and/or avoid blocking, you can't do any better than a carefully configured NeoGPS. It is faster and smaller than all other NMEA libraries. It is even fast enough to run during the RX interrupt, with the special serial classes NeoHWSerial, NeoSWSerial or NeoICSerial.

If you're currently using SoftwareSerial, I hope you realize it's a real CPU killer. If you can use pins 8 & 9 (on a UNO), AltSoftSerial is a better choice. If not, NeoSWSerial is a drop-in replacement that works on any pins and is MUCH more efficient and reliable than SoftwareSerial.

I think you need to decide if you are going to use a console interface or a NMEA sentence interface, and whether you can use a blocking solution (ok) or a non-blocking solution (best).

Cheers,
/dev

[Bryanpl]

zoomkat:
Are you using a converter to convert the rs232 to the arduino TTL format?

Yes

[giovanniguerra]

/dev:
This is not a novice topic, which is unfortunate, because String is so easy to use...

Thanks for the detailed info and link.

[giovanniguerra]

I can't find a way of giving a "thumbs up" or "extra vote" to each individual reply, so thanks all who replied to my "Why Strings are bad" sub thread.