Buffers and arrays for serial data - basic help needed to get started

Using Arduino Programming Questions
1

I have a solar regulator that puts out a 16 byte message which always starts 0xAA.

My goal is to use a Teensy to read that data stream and extract some of the data which I will ultimately display on a small screen. For example byte 4 & 5 contain the solar voltage (LSB & MSB).

I have some basic understanding of the principles which I believe are:

  • Receive the serial data into buffer
  • Define an array
  • Identify the start byte
  • Read the bytes from the buffer into the array
  • Checksum the data
  • Process the relevant bytes
  • Send the resulting data to my display

I can do the basics (I think) which is to receive the data and print the bytes to the serial monitor but where I am struggling is how to identify the start marker (0xAA) so that the array always has the correct 16 bytes within it.

Secondly, once I get the correct bytes into the array how do select specific bytes (such as byte 4 & byte 5 to calcualte the solar voltage?

All help appreciated. My starting attempt at code is below.

/*
Votronic Solar Regulator March 2022
UART reader for output to display Battery Voltage, Solar Voltage and Solar Current

RJ11 6P6C on Votronic MPP165
Pin 1 = Signal
Pin 2 = 5v
Pin 3 = 12v
Pin 4 = 
Pin 5 = 5v
Pin 6 = Ground

1000 Baud; 8 bit; No parity.
Byte 
0 - sync byte 0xAA
1 - ID byte 0x1A
2 - Battery V LSB - U16 10mV/bit
3 - Battery V MSB
4 - Solar V LSB - U16 10mV/bit
5 - Solar V MSB
6 - Solar I LSB - S16 100mA/bit
7 - Solar I MSB
8 - 12 No data
13 - Battery status flags (charge phase - bulk, absorption, float)
14 - Regulator status flags (standby, no solar)
15 - Checksum XOR bytes 1-14
*/

#define solarData Serial1 //set name of data and serial port to use


void setup() {
  solarData.begin(1000,SERIAL_8N1); //set baud and data type
  Serial.begin(1000);
}

void loop() {
  byte incomingByte;
  
  if (solarData.available() > 0)
    incomingByte = solarData.read();
    Serial.print("UART data: ");
    Serial.println(incomingByte, HEX);
    delay(1000);
}

//initial thoughts on start marker code below
/*
void readsolarData() {
  byte incomingByte;
  byte startMarker = 0xAA

  if (solarData.available() > 0) {
    incomingByte = solarData.read();
    if (incomingByte == startMarker)
    
}
*/
2

I'd implement it as a simple state machine.

3

maybe have a look at this tutorial that talks about serial input/output basics.

hope that helps....

4

This is classic serial read & parse.

Thus something like I showed for GPS which starts with '$'

Extract a number from a gsm module text - Software / Arduino SIM - Arduino Forum

5

I am sure there are many ways to code receiving a string like this and someone will come along and tell you the best way, but here is my way.

void pollSerial() {
  static unsigned long serialInTimestamp;// allows us to clear the buffer after too much time
  static unsigned char bytesRead = 0;//count the bytes received from 0-15
  static unsigned char serialBuffer[17] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};//16 bytes + 1 NULL
  if (bytesRead > 0) {// we have received at minimum 0xAA ( aka 170)
    if (millis() - serialInTimestamp > 200UL) {// 160 ms to receive 16 characters at 1000 baud
      memset(serialBuffer, 0, sizeof serialBuffer);// clear the buffer
      bytesRead = 0;// reset count
    }
  }
  if (Serial.available()) {
    int c = Serial.read();// int is what is returned and can also hold characters
    if (bytesRead == 0 && c == 0xAA) {// what we do for 0xAA
      serialInTimestamp = millis();//start the timer
      serialBuffer[bytesRead] = c;// add byte received to the buffer
      bytesRead++;// increment count of bytes received
    }
    else if (bytesRead > 0 && bytesRead < 15UL) {//bytes 1-15
      serialBuffer[bytesRead] = c;// add to buffer(unfilled bytes are NULL)
      bytesRead++;// increment counter
    }
  }
  if (bytesRead >= 16UL) {// we have received the string
    //do something with the string
    memset(serialBuffer, 0, sizeof serialBuffer);// clear the buffer
    bytesRead = 0;// reset count
  }
}
6

If all you want is the value contained in two bytes, it seems pointless to buffer the whole message.
Idle waiting for the 0xAA, then simply count off the bytes until you reach the two you're interested in, and discard the others - just do the XOR on them , and chuck 'em.
Only release the result if the checksum is invalid, otherwise restart the idling.

1 Like
7

Agreed

8

Thanks for the comments so far. I will follow up on each bit of advice, but meantime just to confirm that of the 16 bytes of data I will be needing data contained in at least 8 of them.

9

Still no real need to buffer all of them.

10

Ok, thanks. Is there any chance you could help me on my way with the basics of how to watch out for the 0xAA byte and then count off the following bytes.

11

I wrote the following for a similar application.
It's part of an add-on for an IKEA PM2.5 air quality monitor

    static constexpr byte PM25_MSB {5};
    static constexpr byte PM25_LSB {6};


    enum RX_STATE {WAIT_PREAMBLE, 
                   HANDLE_PAYLOAD, 
                   CHECK_MESSAGE} rxState;

template <class Serial_T>
void IKEA_PM25<Serial_T>::resetRx ()
{
  pm25          = invalidReading;
  rxState       = WAIT_PREAMBLE;
  checksum      = 0;
  messageIndex  = 0;
}
/**************************************************** 
 *
 * Protocol details
 *   Controller: 11 02 0B 01 E1
 *   Sensor:     16 11 0B DF1-DF4 DF5-DF8 DF9-DF12 DF13 DF14 DF15 DF16 [CS]
 *   PM2.5 (ug/m^3)= DF3*256 + DF4
 *
*/ 
template <class Serial_T>
void IKEA_PM25<Serial_T>::handleSensor()
{
  uint32_t now = millis ();
  // check here for sensor timeout? ToDo
  while (serial.available()) {
    uint8_t rxChar = (uint8_t) serial.read ();  
    checksum += rxChar;

    switch (rxState) {
      case WAIT_PREAMBLE:
        if (rxChar == preamble [messageIndex++]) {
          if (messageIndex == preambleLength)
            rxState = HANDLE_PAYLOAD;
        } 
        else 
          resetRx ();
      break;

      case HANDLE_PAYLOAD: // we've had a valid preamble, all we can do is grab / ignore the rest
        if (messageIndex == PM25_MSB)
          pm25 = rxChar << 8;
        if (messageIndex == PM25_LSB)
          pm25 |= rxChar;
        messageIndex++;
        if (messageIndex == messageSize)
          rxState = CHECK_MESSAGE;
      break;

      case CHECK_MESSAGE:
        if (checksum == 0) {
          sensorOnline = true;

          rollingSum -= samples [sampleIndex];
          rollingSum += samples [sampleIndex++] = pm25;
      
          if (sampleIndex >= nSamples) {
            wrapped     = true;
            sampleIndex = 0;
          }
          int divisor = wrapped ? nSamples : sampleIndex;
          avgPM25 = rollingSum / divisor;
          lastReading = now;
        }
        resetRx ();        
      break;
      default:
      break;
    }
  }
}

Ignore the template stuff, it's just a fudge to allow soft or hard UARTs.
Here, the checksum is simply that, not an XOR, and it's over all of the message.

12

Thanks very much. I'll study that and adapt it to my needs.

13

Untitled
Untitled1054×617 321 KB

looking at the frame content, IMHO while '0xAA' is the starter marker, it is not a reserved value ie it can also occur within the message frame data.

Bearing that in mind, this is my take on how I would try to ensure that I receive/read a complete frame as and when it is fully received:
(Compiles, Not tested!)

/*
  Votronic Solar Regulator March 2022
  UART reader for output to display Battery Voltage, Solar Voltage and Solar Current

  RJ11 6P6C on Votronic MPP165
  Pin 1 = Signal
  Pin 2 = 5v
  Pin 3 = 12v
  Pin 4 =
  Pin 5 = 5v
  Pin 6 = Ground

  1000 Baud; 8 bit; No parity.
  Byte
  0 - sync byte 0xAA
  1 - ID byte 0x1A
  2 - Battery V LSB - U16 10mV/bit
  3 - Battery V MSB
  4 - Solar V LSB - U16 10mV/bit
  5 - Solar V MSB
  6 - Solar I LSB - S16 100mA/bit
  7 - Solar I MSB
  8 - 12 No data
  13 - Battery status flags (charge phase - bulk, absorption, float)
  14 - Regulator status flags (standby, no solar)
  15 - Checksum XOR bytes 1-14
*/

#define solarData Serial1 //set name of data and serial port to use
#define PREAMBLE 0xAA
#define PAYLOAD 1
#define CHECKSUM 15
#define solarData_SIZE 16

uint8_t new_preamble = 0;
uint8_t fsm = PREAMBLE;
uint8_t i = 0;

uint8_t data[16];
void setup() {
  solarData.begin(1000, SERIAL_8N1); //set baud and data type
  Serial.begin(1000);
}

void loop() {
  uint8_t temp;

  if (solarData.available()) {
    temp = solarData.read();
    if (temp == PREAMBLE) {
      if (fsm == PREAMBLE) {
        fsm = PAYLOAD;
        new_preamble = 0;
        i = 0;
      }
      else if (new_preamble == 0) {
        new_preamble = i;
      }
    }

    if (fsm == PAYLOAD ) {

      data[i++] = temp;
      if (i > CHECKSUM) {
        fsm = CHECKSUM;

        if (data[0] == PREAMBLE) {
          unsigned char checksum = 0;
          for (char k = 1; k < CHECKSUM; ++k) {
            checksum = ~(checksum ^ data[k]);
          }

          if (checksum == data[CHECKSUM]) {
            Serial.println("Valid DATA: ");
            for (char k = 0; k < solarData_SIZE; ++k) {
              Serial.print(data[k], HEX);
              Serial.print(" ");
            }
            Serial.print("\n");
            new_preamble = 0;
          }
          /*--------------------------------
            else {
            for (char k = 0; k < solarData_SIZE; ++k) {
              printf("%.02X ", data[k]);
            }
            printf("Checksum %.02X\n ", checksum);
            }
          */ // --------------------------------
        }

        if (new_preamble > 0) {
          memmove (&data[0], &data[new_preamble], solarData_SIZE - new_preamble);
          i = solarData_SIZE  - new_preamble;
          fsm = PAYLOAD;
        }
        else {
          fsm = PREAMBLE;
        }
      }
    }
  }
}

hope that helps...

14

Wow, thanks sherzaad, that's a great help. I hadn't considered the implications of other bytes also being 0xAA but that's very possible.

15

you may be interested in code for reading an Automess geiger counter

this approach uses an idx to keep track of not only the next location in the buffer, but whether it's looking for the start byte. bytes are discarded when looking for start and not finding it.

once found, the code looks for the last byte which depends on the length of the msg. in your case, i believe it depends on the msg type which if the 2nd byte.

presumably if the value of the start byte is part of a msg, the msg ID won't be valid nor the checksum. several msgs may pass before synchronized

16

Thanks for the input gcjr. I've realised that both the first and second byte are always the same, so if I can use them both as a start marker that should reduce the chance of the msg holding the same 2 mathcing bytes and confusing the sync.

17

Verifying the checksum as well will make it highly unlikely that you were mislead into starting to read halfway through a packet.

18

i believe looking for 2 start markers is more complicated. better off thinking of it as an invalid msg-ID

consider


byte inp [] = {
    0xAA, 0x1A, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
            0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0xBE,     // valid chksm
    0x00, 0x12, 0x34,
    0xAA, 0x1A, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
            0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0xBE,     // valid chksm
    0xAA, 0x1b, 0x12, 0x34, 0x56,
    0xAA, 0x1A, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
            0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0xBE,     // valid chksm
};
unsigned inpIdx;

#define STX     0xAA
#define MsgId   0x1A
#define MsgLen  16

byte buf [80];
int  idx;
int  len;

char s [80];

// -----------------------------------------------------------------------------

// -----------------------------------------------------------------------------
int
procMsg (void)
{
    int res = 0;

    if (sizeof(inp) > inpIdx)  {
        byte c = inp [inpIdx++];

#if 1
        sprintf (s, "  %s: 0x%02x %2d", __func__, c, idx);
        Serial.println (s);
#endif

        if (0 == idx && STX != c)  {
#ifdef DBG
            Serial.println ("Error STX");
#endif
            return 0;
        }

        if (1 == idx)  {
            if (MsgId == c)
                len = MsgLen;
            else {
                Serial.println ("Error bad ID");
                idx = 0;
                return 0;
            }
        }

        buf [idx++] = c;

        if (0 < len && len <= idx)  {
            byte sum = 0;
            for (int n = 0; n < len; n++)  {
                sum ^= buf [n];
#ifdef DBG
                sprintf (s, " %2d 0x%02x 0x%02x", n, buf [n], sum);
                Serial.println (s);
#endif
            }

            if (0 == sum)  {
                res = idx;
            }
            else  {
                sprintf (s, "%s: Error cksm 0x%02x", __func__, sum);
                Serial.println (s);
            }
            len = idx = 0;
        }
    }

    return res;
}

// -----------------------------------------------------------------------------
void
dispMsg (
    byte  *buf,
    int    nByte )
{
    sprintf (s, "%s: %d -", __func__, nByte);
    Serial.println (s);

    for (int n = 0; n < nByte; n++)  {
        sprintf (s, " 0x%02x", buf [n]);
        Serial.print (s);
    }

    Serial.println ();
}

// -----------------------------------------------------------------------------
void
loop (void)
{
    int n = procMsg ();
    if (n)
        dispMsg (buf, n);
}

// -----------------------------------------------------------------------------
void
setup (void)
{
    Serial.begin (9600);
}
19

I tried running your sketch this evening and it is working a treat. I get a nice stream of hex messages on the serial monitor all starting AA. Now to extract the bytes I want and calculate the info. Many thanks.

20

Check this
See if it helps