Reading all data coming in from Serial port into one char array.

Projects Programming
1

Hello,

I'm struggling to receive all data from a serial port and store it into a char array so I can work with it.
The full data i should receive is:

+QMTRECV: 0,3,"command/test","{"data":"Test","ispublic":true,"ts":1605188045123}"

This is coming in from a quectel BG96 when subscribed to a mqtt topic, so I don't know when it is coming in, I have to catch it, also the length of the "data" could be much higher.

I do receive this when I use Serial.readString(), but since arduino cannot handle strings easily I do not prefer to use this function, I already see it crashing sometimes after running a while because of this.

So I replaced it with the code below:

 uint16_t index = 0;
  char _buffer[255];
  while (cellular.available())
  {
      char c = cellular.read();
      if (c == '\r')
      {
          continue;
      }
      if (c == '\n' && index == 0)
      {
          // Ignore first \n.
          continue;
      }
      _buffer[index++] = c;        
  }
  _buffer[index] = 0;  
  
  if(index > 0)
  {
    loggerPrint("[QUECTEL] SERIAL: ");
    loggerPrintln(_buffer);
  }

Located in the main loop.

This is basicly how all examples I could found show me, I tried some different ways but the outcome is never the right.
This code results the following:

+QMTRECV: 0,1,"command/test","{"data":"Test","ispublic":tru

I figured as this are about 63 bytes, it has something to do with the serial buffer of max 64, but since I'm reading it in the while it should read the bytes before the buffer is full, right?

Thanks!

2

Take a look at Serial input basics - updated

3

I figured as this are about 63 bytes, it has something to do with the serial buffer of max 64, but since I'm reading it in the while it should read the bytes before the buffer is full, right?

Probably.

Serial data is MUCH slower than what the processor can deal with. Receive a byte or 2 and immediately save in a buffer big enough to hold all the data. The serial buffer is just to catch the data somewhere while your code gets round to doing something with it.

4

If you're dropping characters after 63, it does indeed suggest the buffer filled up and the rest were thrown away.
Which in turn makes me think that you're not checking for input frequently enough.

Also, you're trying to read the whole string in one pass, so once you run out of data, you stop. Your reader should be called repeatedly until you have what you need.

All of which is a long winded way of saying check out Robin2's tutorial that UKHeliBob linked.

5

Thank you all for replying.

I already check the tutorial, I use that code and it fails to read more then 64 still.
Also now I testen by reporting the cycle time:

15:40:36.709 -> Time: 6
15:40:36.709 -> Time: 7
15:40:36.709 -> Time: 6
15:40:36.742 -> Time: 6
15:40:36.742 -> Time: 6
15:40:36.742 -> Time: 7
15:40:36.742 -> Time: 6
15:40:36.775 -> [QUECTEL] SERIAL: +QMTRECV: 0,4,"command/test","{"data":"Test","ispublic":true,"
15:40:36.841 -> Time: 92
15:40:36.878 -> [QUECTEL] SERIAL: ts":1605192035721}"
15:40:36.917 -> 
15:40:36.917 -> Time: 47
15:40:36.917 -> Time: 6
15:40:36.917 -> Time: 6
15:40:36.944 -> Time: 6

Each 6 milliseconds the function is called, when it receives data it takes about 92 + 42 seconds to read the data.
So now looking at this it looks like I'm going to fast, the buffer is empty so the while stops, next cycle the buffer contains new data and it catches is, so you I need to have a begin and endpoint, problem is only that I cannot change the serial format coming in.

6

I use that code and it fails to read more then 64 still.

Then you are not using Robin's method correctly

+QMTRECV: 0,3,"command/test","{"data":"Test","ispublic":true,"ts":1605188045123}"

Does the message always start with a "+" and end with a "}" without either a "+" or a "}" appearing in the data itself ?

Please post what you tried that failed

7

Is it possible to post more (or all) of your code or at least a minimally-viable program that illustrates the problem?

Snippets help but in cases like this sometimes other factor contribute to a failure.

8

Do you know the maximum possible length of the message, and is there a specific terminating character, such as a carriage return or linefeed?

9

UKHeliBob:
Then you are not using Robin's method correctly

+QMTRECV: 0,3,"command/test","{"data":"Test","ispublic":true,"ts":1605188045123}"

Does the message always start with a "+" and end with a "}" without either a "+" or a "}" appearing in the data itself ?

Please post what you tried that failed

I did not change anything yet, I checked the Example 2 in his tutorial, that is exactly what I do, only I did not implement the endmarker part as I don't receive my endmarker because I'm not receiving all data, correct me if I'm wrong.
I don't receive this line different times in a row quickly behind each other, so I need split different lines with a endmarker. I want to get all data, process it and wait for a new message to come in.
Thanks.

10

Blackfin:
Is it possible to post more (or all) of your code or at least a minimally-viable program that illustrates the problem?

Snippets help but in cases like this sometimes other factor contribute to a failure.

My file exceeds the max character post length, it's also not very cleaned yet, but that is why I inplemented the cycle time reporter, so it gives me 6 milliseconds on each cycle, could that be the problem?
Other than that, nothing is really running when receiving the serial data.

11

Hi,

So I made a sketch with the same code only nothing else;

#define cellular Serial3
#define logger Serial

void setup() {
  // initialize both serial ports:
  logger.begin(9600);
  cellular.begin(115200);

  logger.write("Command Tester");

  digitalWrite(24, HIGH);
}

void loop() {


  uint16_t index = 0;
  char _buffer[255];
  while (cellular.available())
  {
      char c = cellular.read();
      if (c == '\r')
      {
          continue;
      }
      if (c == '\n' && index == 0)
      {
          // Ignore first \n.
          continue;
      }
      _buffer[index++] = c;     
  }
  _buffer[index] = 0;  
  
  if(index > 0)
  {
    logger.print("[QUECTEL] SERIAL: ");
    logger.println(_buffer);
  }
  /*
  while (cellular.available()) {
    logger.write(cellular.read());
  }
  */
  while (logger.available()) {
    cellular.write(logger.read());
  }
}

//Serial.write(0x1a);
// ALT + 026 in notepad CTRL+C to terminal monitor for CTRL+Z command

The commented while > print is working fine, but when I try to save it to a char array I again dont get more then 64 bytes..

12

I wrote this to simulate a modem transmitting out Serial3 and the Mega receiving on Serial1. This seems to work here but it is just a simulation.

Can you adapt the RX state machine to your minimal code and try it? I'll take your RX code and try it on my sim to see if it works.

//Target: Mega2560

//message sent out Serial3 by modem simulator
char *txMessage = " +QMTRECV: 0,3,\"command/test\",\"{\"data\":\"Test\",\"ispublic\":true,\"ts\":1605188045123}\" ";

#define RX_BUFF_SIZE    100     //will work for test message; may need to be bigger for others

enum estatesRX
{
    ALIGN = 0,
    RX_MSG
};

enum estatesTX
{
    WAIT_TX = 0,
    XMIT_MSG
};

HardwareSerial *spoofModem = (HardwareSerial *)&Serial3;        //whatever port you're using
HardwareSerial *cellular = (HardwareSerial *)&Serial1;          //whatever port you're using
HardwareSerial *console = (HardwareSerial *)&Serial;            //replace with your "loggerPrint"?

//receive buffer
char
    rxBuffer[RX_BUFF_SIZE];
        
void setup() 
{
    //your setup
    console->begin(115200);         //in place of your "loggerPrint"
    spoofModem->begin(9600);        //for bench testing w/out modem (transmits msg at regular intervals
                                    //  for use, connect Serial3 TX to Serial1 RX
    cellular->begin(9600);          //or whatever your baud rate is

}//setup

void loop()
{
    ModemTransmit();                //simulate modem
    ReadModemSerial();              //receive state machine

}//loop

void ReadModemSerial( void )
{
    static uint8_t
        indexRX,
        stateRX = estatesRX::ALIGN;

    if( cellular->available() > 0 )
    {
        while( cellular->available() > 0 )
        {
            uint8_t rxChar = cellular->read();
            switch( stateRX )
            {
                case    estatesRX::ALIGN:
                    //look for a '+' indicating start of message
                    if( rxChar == '+' )
                    {
                        indexRX = 0;
                        rxBuffer[indexRX++] = rxChar;
                        stateRX = estatesRX::RX_MSG;
                        
                    }//if
                
                break;
        
                case    estatesRX::RX_MSG:
                    //rx each new char into buffer but don't overrun end of buffer
                    rxBuffer[indexRX++] = rxChar;
                    if( indexRX == RX_BUFF_SIZE - 1 )
                        indexRX--;

                    //I believe the message is terminated by a "\r\n" sequence; look for the '\n' to indicate the end
                    if( rxChar == '\n' )
                    {
                        rxBuffer[indexRX] = '\0';               //NULL-terminate the message
                        console->print("[QUECTEL] SERIAL: ");   //print it to the console
                        console->println( rxBuffer );
                        
                        stateRX = estatesRX::ALIGN;             //ready for the next message
                                                
                    }//if
                
                break;
                
            }//switch
            
        }//while
        
    }//if
    
}//ReadModemSerial

void ModemTransmit( void )
{
    static uint8_t
        stateTX = estatesTX::WAIT_TX,
        txIndex = 0;
    static uint32_t
        timeCharacter,
        timeMessage;
    uint32_t
        timeNow;

    switch( stateTX )
    {
        case    estatesTX::WAIT_TX:
            timeNow = millis();
            if( (timeNow - timeMessage) >= 1000ul )
            {
                timeMessage = timeNow;
                timeCharacter = micros();
                txIndex = 0;
                stateTX = estatesTX::XMIT_MSG;
                
            }//if
            
        break;
        
        case    estatesTX::XMIT_MSG:
            timeNow = micros();
            if( (timeNow - timeCharacter) >= 2000ul )   //write a char every 2mS
            {
                timeCharacter = micros();
                
                uint8_t txChar = (uint8_t)*( txMessage + txIndex++ );
                if( txChar != '\0' )
                    spoofModem->write( txChar );
                else
                {
                    spoofModem->write( '\r' );    
                    spoofModem->write( '\n' );
                    stateTX = estatesTX::WAIT_TX;
                }
                
            }//if                    
                        
        break;
        
    }//switch

}//for
13

Hi thank you for your reply.

As most times you're right I was wrong.
I copied the code from the Tutorial Example 2.
And magical now I receive the full string.

I removed some stuff and it is working okay now;

const byte numChars = 255;
    char receivedChars[numChars];   // an array to store the received data
    static byte ndx = 0;
    char c;
   
    while (cellular.available() > 0) {
        c = cellular.read();

        if (c != '\n') {
            receivedChars[ndx] = c;
            ndx++;
            if (ndx >= numChars) {
                ndx = numChars - 1;
            }
        }
        else {
            receivedChars[ndx] = '\0'; // terminate the string
            ndx = 0;
            logger.println(receivedChars);
        }
    }

I guess it is because of the endmarker, but I still don't understand why all data is coming in now and not before.