Build array from serial read of unknown length and without terminating character

[Zimbu]

Based on some guidance received here, I've previously approached this problem using serial.find plus parse.int, BUT now I realize that solution doesn't work if I am looking for two different targets...So allow me to restate the problem:

I send a command to a bluetooth LE module, and receive back a serial response. For example, this:

OK+DISISOK+DISC:00000000:00000000000000000000000000000000:0000000
000:B8782E07067C:-074OK+DISC:4C000215:2F234454CF6D4A0FADF2
F4911BA9FFA6:00000001AC:0CF3EE041CCE:-046OK+DISC:65000337:2RG37784CF6D4A0FAD
F2F5688BA9FFA9:00000001AC:0DS3EW041BNE:-082OK+DISCE

(the response is one long string; in the above example, I split it into several lines and added color so it is easier to read in this forum)

In this example, the module has discovered 3 different bluetooth LE beacons in the room:

B8782E07067C with rssi value of -74 (the integer that follows directly after B8782E07067C)
0CF3EE041CCE with rssi value of -46
0DS3EW041BNE with rssi value of -82

I want to scan the response and store the rssi values for specific target beacons B8782E07067C and 0DS3EW041BNE but ignore everything else.

Complicating factors/requirements/helpful points:

1. I want to use an array and not use a String (capital S).

2. There is no terminating character on the data received from the module. But, it always ends with "OK+DISCE", so maybe that's good news?

3. The data received from the module is NOT of fixed length. For example, if there were 5 beacons in the room, the data response string would be longer. If there were 2 beacons, shorter. The module has no way of telling me in advance how many beacons are in the room. In other words, the length of the data received from the module can not be determined in advance. The order of the beacons in the data string is also variable.

4. The full data string always starts with "OK+DISIS". Eight characters.

5. Each beacon's info in the string is proceeded by "OK+DISC:", which in effect means, "hey, I discovered a beacon and here's its 70 character info".

6. The length of each beacon's substring data is always exactly 70 characters long, not including the OK+DISC: at the beginning. If you include the OK+DISC:, then each beacon's data substring is 78 characters long.

7. Summary of total data string received:
   -always starts with same 8 characters, "OK+DISIS"
   -each beacon's info substring always starts with same 8 characters, "OK+DISC:"
   -each beacon's info substring contains 70 characters, not including the "OK+DISC:"
   -each beacon's info substring contains the important identifying 12 character ID number from characters 54 to 65
   -after the 12 character ID number of each beacon, there is a colon, then the negative integer that I want to store is located in characters 67 to 70

8. To clarify bit more, here's the same example data string, but I have broken it down and added notes to make it easier to understand:

8 characters: OK+DISIS

78 characters with ID number from characters 54 to 65 and rssi number from characters 67 to 70: OK+DISC:00000000:00000000000000000000000000000000:0000000000:B8782E07067C:-074

78 characters with ID number from characters 54 to 65 and rssi number from characters 67 to 70: OK+DISC:4C000215:2F234454CF6D4A0FADF2F4911BA9FFA6:00000001AC:0CF3EE041CCE:-046

78 characters with ID number from characters 54 to 65 and rssi number from characters 67 to 70: OK+DISC:65000337:2RG37784CF6D4A0FADF2F5688BA9FFA9:00000001AC:0DS3EW041BNE:-082

...maybe one or two more 78 character substrings here for other beacons in the room...

8 characters: OK+DISCE

Sample pseudo code:

void loop() 
{
  
// read the input from the Bluetooth LE 4.0 scanner module

  if (millis() - timestamp1 > 500) //Time delay between scans for beacons is 0.5 seconds; this prevents the program from sending too many requests for scan and overwhelming the Bluetooth LE 4.0 scanner module
  {
    BluetoothScanner.write("AT-DISI?"); // ask HM-10 to scan for bluetooth LE 4.0 beacons
    timestamp1 = millis(); // reset the countdown timer for delay between scans
  }
  
  if (BluetoothScanner.available()) // If there is a message from the HM-10, then do what follows in brackets
  {
    ...Scan the incoming message from bluetooth module to see if it contains the target beacons, B8782E07067C and 0DS3EW041BNE; if it does, then capture the integers following each and store those values in variables...

      Serial.print("The rssi for B8782E07067C: ");
      Serial.println(rssiTarget1);
      Serial.print("The rssi for 0DS3EW041BNE: ");
      Serial.println(rssiTarget2);
      
   }
    
}

My rough idea is something like:

-Data string comes in from the module
-Count total characters in string to figure out number of beacons discovered, e.g. 328 characters received means 4 beacons in the room (8 characters at start, 8 characters at end, and 4 sets of 78 characters in between).
-Make a separate array for each of these sets of 78 characters
-Check characters # 54 to 65 for each array to see if it is one of the target beacons in which I'm interested
-If so, store the target ID number and its associated rssi integer (located from characters 67 to 70)
-Do something with those rssi integers
-Clear out all this info
-Loop again

Sorry for the wordy description. Thanks in advance for any ideas!

[Ps991]

Serial monitor baud rate to 115200. Add general variables, setup stuff to the program

//includes

//variables
const int     timeout   = 3000; //3 seconds is long enough?
unsigned long timeStamp = 0;    //a timing variable

char buffer[78]  = {}; //stores our Serial received characters
byte bufferCount = 0;  //used to keep track of where we store the next Serial received character

const char responseStartSeq[] = "OK+DISIS"; //the response start sequence
const char beaconStartSeq[]   = "OK+DISC:"; //the beacon start sequence

byte          stage             = 0;   //which state are we currently in
byte          stageCompareCount = 0;   //a variable used differently in each state
unsigned long stageTimeStamp    = 0;   //holds timing info for each state
int           stageTimeout      = 200; //200ms timeout

int rssi = 0; //we will store the rssi in here

enum Stage { //an enumeration
  RESPONSE_START = 0,
  BEACON_START,
  BEACON_INFO,
  BEACON_ID,
  BEACON_RSSI
};

void setup() {
  Serial.begin(115200);
}

void loop() {
  // read the input from the Bluetooth LE 4.0 scanner module
  if (millis() - timeStamp > 5000) { //Time delay between scans for beacons is 5 seconds; this prevents the program from sending too many requests for scan and overwhelming the Bluetooth LE 4.0 scanner module
    Serial.println("Starting...");
    while (Serial.available())
      Serial.read(); //clear out the serial buffer before we start, just in case
    BluetoothScanner.write("AT-DISI?"); // ask HM-10 to scan for bluetooth LE 4.0 beacons
    timeStamp = millis(); // reset the countdown timer for delay between scans

    boolean failed = false;
    
    while (millis() - timeStamp < timeout && !BluetoothScanner.available()); //wait for response or timeout
    if (millis() - timeStamp >= timeout) {
      Serial.println("BT LE took too long to respond");
      failed = true;
    }

    bufferCount = 0; //set the index for 'buffer' to 0, otherwise we will overflow it
    stageCompareCount = 0; //always set this to 0 when changing states
    stage = RESPONSE_START; //set the state to the very first state after a requested scan
    stageTimeStamp = millis(); //keeps track of the start time, so we can know if there is a timeout
    boolean completeResponse = false;
    while (!failed && BluetoothScanner.available()) { // If there is a message from the HM-10, then do what follows in brackets
      buffer[bufferCount++] = Serial.read(); //read the new data into our buffer

      switch (stage) { //execute certain code depending on the current state/stage
        case RESPONSE_START: //this is the very first state after requesting a scan
          if (buffer[bufferCount - 1] == responseStartSeq[stageCompareCount]) { //compare the received data with the response start sequence that the BT LE device sends back
            stageCompareCount++; //this i used to keep track of how many characters have matched the response start sequence
            if (stageCompareCount == sizeof(responseStartSeq) - 1) { //the entire sequence has match, move on to the next state
              stage = BEACON_START; //next state
              stageCompareCount = 0;
              stageTimeStamp = millis();
              bufferCount = 0; //reset back to 0, now we treat all new data as detected devices
              
              Serial.print("Got BT LE start response: ");
              for (int i = sizeof(responseStartSeq) - 1 - 1; i >= 0; i--)
                Serial.print(buffer[7 - i]); //print it
              Serial.println();
            }
          } else {
            Serial.println("Error: RESPONSE_START sequence did not match");
          }

          //implement timeout if not reach 4 in reasonable time
          if (millis() - stageTimeStamp >= stageTimeout) {
            failed = true;
            Serial.println("Timeout at stage: RESPONSE_START!");
            break;
          }
          break;
        case BEACON_START:
          if (buffer[bufferCount - 1] == beaconStartSeq[stageCompareCount]) { //compare received data with beacon start sequence
            stageCompareCount++;
            if (stageCompareCount == sizeof(beaconStartSeq) - 1) { //the sequence has been matched, move on to next state
              stage = BEACON_INFO;
              stageCompareCount = 0;
              stageTimeStamp = millis();
              
              Serial.print("Got beacon start response: ");
              for (int i = sizeof(beaconStartSeq) - 1 - 1; i >= 0; i--)
                Serial.print(buffer[7 - i]); //print it
              Serial.println();
            }
          } else {
            Serial.println("Error: BEACON_START sequence did not match");
          }

          //implement timeout if not reach 4 in reasonable time
          if (millis() - stageTimeStamp >= stageTimeout) {
            failed = true;
            Serial.println("Timeout at stage: BEACON_START!");
            break;
          }
          break;
        case BEACON_INFO:
          stageCompareCount++; //keeps track of how many characters were received
          if (stageCompareCount == 53) { //wait for 53 characters. If this is true, then move onto next state
            if (buffer[16] != ':' || buffer[49] != ':' || buffer[60] != ':') { //an error check, colons are expected to be at these locations, if they are not then there is an issue
              Serial.println("Error: Unexpected syntax at stage: BEACON_INFO");
              failed = true;
              break;
            }
            stage = BEACON_ID;
            stageCompareCount = 0;
            stageTimeStamp = millis();
            
            Serial.print("Got beacon info: ");
            for (int i = 53 - 1; i >= 0; i--)
              Serial.print(buffer[60 - i]); //print it
            Serial.println();
          }

          //implement timeout if not reach 53 in reasonable time
          if (millis() - stageTimeStamp >= stageTimeout) {
            failed = true;
            Serial.println("Timeout at stage: BEACON_INFO!");
            break;
          }
          break;
        case BEACON_ID: //pretty much the same thing as 'BEACON_INFO'
          stageCompareCount++;
          if (stageCompareCount == 13) {
            if (buffer[73] != ':' ) {
              Serial.println("Error: Unexpected syntax at stage: BEACON_ID");
              failed = true;
              break;
            }
            stage = BEACON_RSSI;
            stageCompareCount = 0;
            stageTimeStamp = millis();
            
            Serial.print("Got beacon id: ");
            for (int i = 13 - 1; i >= 0; i--)
              Serial.print(buffer[73 - i]); //print it
            Serial.println();
          }

          //implement timeout if not reach 13 in reasonable time
          if (millis() - stageTimeStamp >= stageTimeout) {
            failed = true;
            Serial.println("Timeout at stage: BEACON_ID!");
            break;
          }
          break;
        case BEACON_RSSI:
          stageCompareCount++;
          if (stageCompareCount == 4) {
            Serial.print("Got beacon rssi: ");

            rssi = (buffer[75] - '0') * 100 + (buffer[76] - '0') * 10 + (buffer[77] - '0'); //convert ASCII text to an integer
            if (buffer[74] == '-') //factor in the negative if neccessary
              rssi = -rssi;
            Serial.println(rssi); //print it

            completeResponse = true; //This signals the code to start a new search for the next device
          }

          //implement timeout if not reach 4 in reasonable time
          if (millis() - stageTimeStamp >= stageTimeout) {
            failed = true;
            Serial.println("Timeout at stage: BEACON_RSSI!");
            break;
          }
          break;
        default:
          break;
      } //end switch

      if (failed) {
        Serial.println("Scan Cancelled!");
        break;
      }

      //need to wait for the next character to arrive
      //Serial is much slower than your code can execute, so we wait
      timeStamp = millis(); // reset the timestamp for received characters
      while (millis() - timeStamp < stageTimeout && !BluetoothScanner.available()); //wait for response or timeout
      if (millis() - timeStamp >= stageTimeout || !BluetoothScanner.available()) {
        if (completeResponse)
          break; //it timed out because there is no more data to send, no need to post an error
        failed = true;
        Serial.println("Error: Receive rate was too slow or data stopped transmitting!");
        break;
      }

      if (completeResponse) { //device info has been completely found, reset the variables and look for another device
        Serial.println("Response Complete!");
        bufferCount = 0;
        stageCompareCount = 0;
        stage = BEACON_START; //there is only 1 RESPONSE_START per scan, so we don't need to look for it again
        stageTimeStamp = millis();
        completeResponse = false;
      }
    }
  }
}

[sterretje]

Is your first record corrupt? It's one shorter than the other two (missing a colon).

OK+DISIS
OK+DISC:00000000:00000000000000000000000000000000:0000000000B8782E07067C:-074
OK+DISC:4C000215:2F234454CF6D4A0FADF2F4911BA9FFA6:00000001AC:0CF3EE041CCE:-046
OK+DISC:65000337:2RG37784CF6D4A0FADF2F5688BA9FFA9:00000001AC:0DS3EW041BNE:-082
OK+DISCE

I think

first wait for OK or OK+
next wait for +DISIS or DISIS (depending on above)

next
1
wait for OK or OK+
2
next wait for +DISC or DISC
3
start counting colons (4 colons; read a char at a time)
4
read till next OK or OK+ and store in buffer
5
read next 6 (5) characters into a buffer
6
if +DISCE or DISCE, your done
else if +DISC or DISC, go to 3

The alternative is like Ps991 suggests; read the full 'line' into a buffer (hits buffer array might have to be one character longer).

[nickgammon]

A state machine would seem appropriate here.

Gammon Forum : Electronics : Microprocessors : State machines

Build array from serial read of unknown length ...
...
The length of each beacon's substring data is always exactly 70 characters long

So you do know the length.

The state machine can initialize a state when it gets "OK+DISC:" and then process the next 70 characters. Sounds simple enough. Don't bother trying to read the whole lot (how much RAM do you have?). Just process one device at a time.

[Robin2]

Looking at the data in the Original Post I suggest using the second example in Serial Input Basics, but with the end-marker changed to a colon. That should give you output like this

000B8782E07067C
-074OK+DISC
4C000215
2F234454CF6D4A0FADF2F4911BA9FFA6
00000001AC
0CF3EE041CCE
-046OK+DISC
65000337
2RG37784CF6D4A0FADF2F5688BA9FFA9
00000001AC
0DS3EW041BNE

If there is then an appreciable delay at the end of the transmission you could use that to confirm that you have all the data.

...R

[Ps991]

After testing, I get

Starting...
Got BT LE start response: OK+DISIS
Got beacon start response: OK+DISC:
Got beacon info: 0000000:00000000000000000000000000000000:0000000000:
Got beacon id: B8782E07067C:
Got beacon rssi: -74
Response Complete!
Got beacon start response: OK+DISC:
Got beacon info: C000215:2F234454CF6D4A0FADF2F4911BA9FFA6:00000001AC:
Got beacon id: 0CF3EE041CCE:
Got beacon rssi: -46
Response Complete!
Got beacon start response: OK+DISC:
Got beacon info: 5000337:2RG37784CF6D4A0FADF2F5688BA9FFA9:00000001AC:
Got beacon id: 0DS3EW041BNE:
Got beacon rssi: -82
Response Complete!
Error: BEACON_START sequence did not match
DEBUG: INFINITE LOOP

The error is expected b/c of the way I tested it

[PieterP]

If you use a string (array of chars), you can try this:

char *data = "OK+DISISOK+DISC:00000000:00000000000000000000000000000000:0000000000:B8782E07067C:-074OK+DISC:4C000215:2F234454CF6D4A0FADF2F4911BA9FFA6:00000001AC:0CF3EE041CCE:-046OK+DISC:65000337:2RG37784CF6D4A0FADF2F5688BA9FFA9:00000001AC:0CF3EE041CCE:-082OK+DISCE";
char *end;
char* tmp = strstr(data,"OK+DISC:");
while (tmp != NULL) {
  // printf("%12lx\t",strtol(&tmp[61],&end,16));
  Serial.print(strtol(&tmp[61],&end,16),HEX); // MAC
  Serial.print('\t');
  // printf("%ld\n",strtol(&tmp[74],&end,10)); // RSSI
  Serial.println(strtol(&tmp[74],&end,10));
  tmp++;
  tmp = strstr(tmp,"OK+DISC:");
}

That's a little more compact

[MarkT]

Did you use cut and paste? Some of the hex strings contain non-hex characters like W and R, I wonder
if that's corruption on the serial line that you need to sort out if you want the thing to work reliably? Is the
last field a checksum? If so be sure to check it.

[vaj4088]

What about line terminators such as carriage return and line feed?

[sterretje]

A state machine would seem appropriate here.
Gammon Forum : Electronics : Microprocessors : State machines

That's basically what the first reply implemented.

vaj4088:
What about line terminators such as carriage return and line feed?

OP stated 'one long string'

[Zimbu]

sterretje:
Is your first record corrupt? It's one shorter than the other two (missing a colon).

Oops, I must have deleted a colon by accident when I cut and pasted. I have fixed it in the original post now. Thanks.

[Zimbu]

MarkT:
Did you use cut and paste? Some of the hex strings contain non-hex characters like W and R, I wonder
if that's corruption on the serial line that you need to sort out if you want the thing to work reliably? Is the
last field a checksum? If so be sure to check it.

Yikes, you are watching very closely!

That third beacon data is fake; I "created" it for illustration purposes. I just copied the first beacon and changed a few characters after pasting. So it was me who incorrectly added a W and an R...I had no idea they are non-hex.

[Zimbu]

Thanks everybody for the awesome suggestions!

My wife is having a party today with all her female friends, so I am kicked out of the house for most of the day today. Will start working on your suggestions tomorrow morning first thing.

Thanks again. Very appreciative that you would all take time to offer your thoughts!

[vaj4088]

Yes, one long string but what invisible characters are at the end of the string? It could happen but it is not like a vendor to make interpretation unnecessarily hard.

Hex characters are 0 1 2 3 4 5 6 7 8 9 A B C D E F (or a b c d e f).
Hex in this case is short for hexadecimal (16). Notice that there are exactly 16 hex characters.

[Robin2]

It does not look like "one long string" to me.

It seems to be several strings separated with colons.

There is no essential difference between a colon as a separator and a line-feed character.

...R

[mrburnette]

You may wish to look at how GPS libraries pull this off. I am very familiar with Adafruit's GPS lib by Ladyada: study it and pick up some new techniques.

I've written a few GPS clock decode routines, these type of responses are well suited for a state-machine (as previously suggested.)

Ray

[Zimbu]

vaj4088:
Yes, one long string but what invisible characters are at the end of the string? It could happen but it is not like a vendor to make interpretation unnecessarily hard.

No, as far as I know, there are now invisible characters at the end of the string. As rogerClark said in another thread: "Unfortunately the HM10's AT commands and more importantly what it sends back, don't have a termination character, so its hard to determine when it has finished sending back a response"

[Zimbu]

Robin2:
It does not look like "one long string" to me.

It seems to be several strings separated with colons.

There is no essential difference between a colon as a separator and a line-feed character.

...R

OK, Robin, I went with your method first since the examples in Serial Input Basics are well documented for dumbos like me.

I got nice data in the serial (while scanning only one beacon) such as:

This just in ... OK+DISCEOK-DISISOK-DISISOK-DISISOK+DISC
This just in ... 4C000215
This just in ... 2F234454CF6D4A0FADF2F4911BA9FFA6
This just in ... 00000001A0
This just in ... 0CF3EE093DA4
MATCH
This just in ... -060OK+DISCEOK-DISISOK+DISC
This just in ... 4C000215
This just in ... 2F234454CF6D4A0FADF2F4911BA9FFA6
This just in ... 00000001A0
This just in ... 0CF3EE093DA4
MATCH
This just in ... -059OK+DISCEOK+DISC
This just in ... 4C000215
This just in ... 2F234454CF6D4A0FADF2F4911BA9FFA6
This just in ... 00000001A0
This just in ... 0CF3EE093DA4
MATCH
This just in ... -060OK+DISCEOK-DISISOK-DISISOK-DISISOK-DISISOK+DIS
This just in ... 4C000215
This just in ... 2F234454CF6D4A0FADF2F4911BA9FFA6
This just in ... 00000001A0
This just in ... 0CF3EE093DA4
MATCH

The "MATCH" is generated by simple if statement in the code using:

if(strcmp(receivedChars, "0CF3EE093DA4") == 0)

First of all, is using strcmp still a legit "array" function or have I strayed into the dreaded String (capital S) area by using this? (I've read in these forums many times that capital S Strings are a bad place to go)

Second, assuming it's OK to use strcmp here, how do I get the sketch to do this:

"If one of the target beacons is scanned (e.g. "0CF3EE093DA4" shows up as one of the receivedChars arrays), then the next four characters of the next loop's receivedChars is the RSSI value for that same beacon, and if that RSSI value is X, then do Y"

Thanks!

Full code below (basically, it's your Example 2):

#include <SoftwareSerial.h>

SoftwareSerial HM10(2, 3); //HM10(Receive Pin, Transmit Pin) -This assumes that the BLE RX to pin 2 and TX to pin 3. Any digital pins may be used.

unsigned long timestamp1; //needed for countdown clock for sending scan requests to HM-10 module

const byte numChars = 51;
char receivedChars[numChars];   // an array to store the received data
boolean newData = false;

void setup()
{
  Serial.begin(57600);  // Begin the Serial Monitor connection at 9600bps
  HM10.begin(57600);  // Begin the HM-10 connection at 9600bps
}


// Example 2 - Receive with an end-marker

void loop() 
{
  
  if (millis() - timestamp1 > 500) //Time delay between scans for bluetooth LE 4.0 fob every 0.5 seconds; this prevents the program from sending too many requests for scan and overwhelming the HM-10
  {
    HM10.write("AT-DISI?"); // ask HM-10 to scan for bluetooth LE 4.0 fob
    timestamp1 = millis(); // reset the countdown timer for delay between scans
  }  
  
    recvWithEndMarker();
    showNewData();

    if (Serial.available()) // Read from Serial Monitor and send to HM-10
    HM10.write(Serial.read());
}

void recvWithEndMarker() 
{
    static byte ndx = 0;
    char endMarker = ':';
    char rc;
    
    while (HM10.available() > 0 && newData == false) {
        rc = HM10.read();

        if (rc != endMarker) {
            receivedChars[ndx] = rc;
            ndx++;
            if (ndx >= numChars) {
                ndx = numChars - 1;
            }
        }
        else {
            receivedChars[ndx] = '\0'; // terminate the string
            ndx = 0;
            newData = true;
        }
    }
}

void showNewData() 
{
    if (newData == true) {
        Serial.print("This just in ... ");
        Serial.println(receivedChars);
        newData = false;
    }
    if(strcmp(receivedChars, "0CF3EE093DA4") == 0)
    {
      Serial.println("MATCH");
    }
}

[sterretje]

You did not end up with Strings (upper case S) but with strings (lower case s) so you're OK.

You want to replace

    if(strcmp(receivedChars, "0CF3EE093DA4") == 0)
    {
      Serial.println("MATCH");
    }

by something else?

void loop()
{
  ...
  ...

    if(strcmp(receivedChars, "0CF3EE093DA4") == 0)
    {
      if(strcmp(rssi, "-123") == 0)
      {
        doSomething();
      }
      else
      {
        doSomethingElse();
      }
}

void doSomething()
{
  Serial.println("rssi match");
}

void doSomethingElse()
{
  Serial.println("no rssi match");
}

I leave it up to you to get the rssi out of the received data.

PS
I think you will have to combine your implementation of Robin's code with some form of statemachine (as mentioned before).

[Robin2]

Zimbu:
"If one of the target beacons is scanned (e.g. "0CF3EE093DA4" shows up as one of the receivedChars arrays), then the next four characters of the next loop's receivedChars is the RSSI value for that same beacon, and if that RSSI value is X, then do Y"

When you get a match you need to set a variable (referred to as a state variable) perhaps matchDetected = true;

When you recieve ANY message you need to do something with it if (matchDetected == true) and when you have done that something set matchDetected = false;. Something like

if (newData == true) {
   if (matchDetected == true) {
       // pick out the 4 chars
       matchDetected = false;
   }
}

This relies on the fact that the message with the desired 4 characters ALWAYS follows immediately after the message what has the beacon ID.

The whole thing will be much easier to manage if you move code out of loop() and into small functions.

...R