UART Communication between ESP32 and ATMEGA2560

Hi all. First post, so please excuse any faux pas.

As the title suggests I am having trouble with successfully sending data between ESP32 and ATMEGA2560 via UART.

The "big picture" project I am on involves sending a 16x12 array of numbers from an ESP32 thermal camera unit to a Controllino. My current project is a simplified version of that - I am trying to send 32 random numbers between 0 and 250 as bytes, separated by commas and bound between "<" and ">" from the ESP32 (see first attached code) to the ATMEGA. I want that data stream to be received and displayed in serial monitor as the same numbers with spaces instead of commas. Is that achievable?! Surely!

#define GRIDX 8
#define GRIDY 4

#define RX 16
#define TX 17

byte rawGrid[GRIDX][GRIDY];


void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200, SERIAL_8N1, RX, TX);
}

void loop() {
  // put your main code here, to run repeatedly:
  for (int y = 0; y < GRIDY; y++) { //across columns first, row by row
    for (int x = 0; x < GRIDX; x++) {
      rawGrid[x][y] = random(0, 250);
    }
  }

  Serial.print("<");
  for (int y = 0; y < GRIDY; y++) { //across columns first, row by row
    for (int x = 0; x < GRIDX; x++) {
      Serial.print(rawGrid[x][y]);
      Serial.print(',');
    }
    //    Serial.print('\n');
  }

  Serial.println(">");
  delay(4000);
}

The code that I am using is modified from the excellent Serial Input Basics tutorial.

I have confirmed that that the data is being transmitted correctly at the TX pin through serial monitor and oscilloscope.

I believe the problem exists on the MEGA side. There is some high strangeness going on. In the second attached code, in the section following else if (rb == ',') { uncommenting any of those lines results in changed behaviour of showData(). See the images of the serial outputs from the MEGA with and without the Serial.println(value) line.

#define GRIDX 8
#define GRIDY 4

int myArray[GRIDX * GRIDY];

boolean newData = false;

void setup() {
  Serial.begin(9600);
  Serial1.begin(115200); //Communication between devices
  Serial.println("<Arduino is ready>");
}

void loop() {
  recvBytesWithStartEndMarkers();
}

void recvBytesWithStartEndMarkers() {
  static boolean recvInProgress = false;
  byte startMarker = '<';
  byte endMarker = '>';
  byte rb;
  int value = 0;
  byte ndx = 0;


  while (Serial1.available() > 0 && newData == false) {

    rb = Serial1.read();

    if (recvInProgress == true) {
      if (rb != endMarker) {
        if (rb != ',') {
          value = value * 10 +  (rb - '0');
        }
        else if (rb == ',') {
          myArray[ndx] = value;
          //Commenting out any of the below 3 lines changes behaviour of showData()?!
          Serial.println(value);
          //Serial.println(myArray[ndx]);
          //Serial.println(".");

          ndx++;
          value = 0;
        }
      }
      else if (rb == endMarker) {
        //Serial.println("end marker seen");
        recvInProgress = false;
        ndx = 0;
        newData = true;
        showNewData();
      }
      else {
        Serial.println("Should never be here");
      }
    }

    else if (rb == startMarker) {
      recvInProgress = true;
      //Serial.println("Start marker seen");
    }
  }
}

void showNewData() {
  if (newData == true) {
    Serial.println("Temperature array... ");
    for (int i = 0; i < GRIDY * GRIDX; i++) {
      Serial.print(myArray[i]);
      Serial.print(' ');
    }
    newData = false;
    Serial.println();
  }
}

I found the solution by learning more about the serial buffer and how Serial.available() works.

Essentially the MEGA was reading and processing the data bytes in the buffer faster than they were coming through. As such, the while loop was exiting and 'value' and 'ndx' were repeatedly being reinstantiated. Adding in the Serial.println() acted as a delay and enabled some more data to accumulate in the buffer, so some data was getting through.

Perhaps a janky explanation, but thats noobs for you.

Fixed the problem by moving 'value' 'ndx' and 'rcvInPogress' to global. Such a simple fix to something thats been buggin me for hours!

Out of interest, how much could I juice the baud rate up on Serial1 before encountering issues?

from an ESP32 thermal camera unit to a Controllino

first thought which came in my mind: why does he use a serial connection and not Ethernet/WiFi and sending via TCP.

There are people round here using 500,000 baud between Arduinos. I can't imagine the ESP32 not being up to the same.

noiasca:
first thought which came in my mind: why does he use a serial connection and not Ethernet/WiFi and sending via TCP.

Absolutely! I had already managed to get my data to transmit using the ESP32's WiFi. I was attempting serial communication partially as an academic exercise (my first time using UART for device to device comms), and partially due to the higher reliability of the ethernet connection (at the ATMEGA end) over WiFi.
Thanks for your input

So your trying to read stuff in at 11500baud and expect the reporting\debug at 9600baud to keep up ?

myArray[ndx] = value;
//Commenting out any of the below 3 lines changes behaviour of showData()?!
Serial.println(value);
//Serial.println(myArray[ndx]);
//Serial.println(".");

Well this just the situation that my SafeString library 'solves' Your Serial.print()s are blocking the rest of the loop from running.
See the Serial Text I/O for the Real World for the details
A revised sketch is

#define GRIDX 8
#define GRIDY 4
#include <BufferedOutput.h>  // from SafeString library from Arduino Library manager
createBufferedOutput(output,100,DROP_UNTIL_EMPTY);
int myArray[GRIDX * GRIDY];
boolean newData = false;
void setup() {
  Serial.begin(9600);  // <<<<<<<<<<<< this is slooow
  // the first recommendation in Serial I/O for the Real World is to increase the baud rate
  output.connect(Serial);
  Serial1.begin(115200); //Communication between devices
  Serial.println("<Arduino is ready>");
}
void loop() {
  output.nextByteOut(); // send more buffered bytes
  recvBytesWithStartEndMarkers();
}
void recvBytesWithStartEndMarkers() {
  static boolean recvInProgress = false;
  byte startMarker = '<';
  byte endMarker = '>';
  byte rb;
  int value = 0;
  byte ndx = 0;
  while (Serial1.available() > 0 && newData == false) {
    rb = Serial1.read();
    if (recvInProgress == true) {
      if (rb != endMarker) {
        if (rb != ',') {
          value = value * 10 +  (rb - '0');
        }
        else if (rb == ',') {
          myArray[ndx] = value;
          //Commenting out any of the below 3 lines changes behaviour of showData()?!
          output.println(value);
          //output.println(myArray[ndx]);
          //output.println(".");
          ndx++;
          value = 0;
        }
      }
      else if (rb == endMarker) {
        //output.println("end marker seen");
        recvInProgress = false;
        ndx = 0;
        newData = true;
        showNewData();
      }
      else {
        output.println("Should never be here");
      }
    }

    else if (rb == startMarker) {
      recvInProgress = true;
      //output.println("Start marker seen");
    }
  }
}

void showNewData() {
  if (newData == true) {
    output.println("Temperature array... ");
    for (int i = 0; i < GRIDY * GRIDX; i++) {
      output.print(myArray[i]);
      output.print(' ');
    }
    newData = false;
    output.println();
  }
}

Now you will get all your data and your debug statements will not block the loop. But don't expect the debug output at 9600 to keep up with the 115200 input. Some debug output will be dropped and ~~ shown instead.
See the tutorial for ways to prioritize your output.
Re collecting the input and parsing the data, the Serial Text I/O for the Real World has some other examples (e.g. parsing GPS comma separate data) that I present as clearer and less prone to programmer error.
I suggest you go from < > to \n terminated lines for the data and use the SafeString readUntilToken() to handle all the low level read stuff. Again see the detailed examples in my tutorial.

Just for interest here is a complete sketch for reading and parsing and error checking the data.
It uses SafeStringStream to inject test data for ease of check the error detecting code.

#define GRIDX 8
#define GRIDY 4
#include <SafeString.h> // from SafeString library from Arduino Library manager
#include <SafeStringStream.h> // from SafeString library from Arduino Library manager
#include <BufferedOutput.h>  // from SafeString library from Arduino Library manager
createBufferedOutput(output,100,DROP_UNTIL_EMPTY);

#define TEST_DATA

#ifdef TEST_DATA
char data[] = ",34,35,37,38\n" //partial first line
              "34,55,77,4,5,6,7,8,9,10,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33\n" // correct data
              "a, z, ,,10,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,36\n"  // some corrupted data
              "4,5,6,7,8,9,10,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,36,36,38\n"; // to much data

cSFP(sfData, data); // wrap the data
SafeStringStream dataStream(sfData); // put data in a SafeStringStream
#else
 HardwareSerial& dataStream = Serial1; // for real data
#endif

const size_t maxCmdLength = 160; // GRIDX * GRIDY * 5; // length of largest line to be recognized, can handle longer input but will not tokenize it.
createSafeString(input, maxCmdLength+1); //  to read input cmd, large enough to hold longest cmd + leading and trailing delimiters
createSafeString(dataLine, maxCmdLength+1); // for parsing, capacity should be >= input
bool skipToDelimiter = false; // bool variable to hold the skipToDelimiter state across calls to readUntilToken()
// set skipToDelimiter = true to skip initial data upto first delimiter.
// skipToDelimiter = true can be set at any time to next delimiter.
              
int myArray[GRIDX * GRIDY];

void setup() {
  Serial.begin(115200);  // make the debug out faster then the serial coms
  for (int i=10;i>0;i--) {
    delay(500);
    Serial.print(i);Serial.print(' ');
  }
  Serial.println();
  Serial.println("<Arduino is ready>");
  // the first recommendation in Serial I/O for the Real World is to increase the baud rate
  SafeString::setOutput(Serial); // errors and SafeString debug go directly to Serial and so may BLOCK!!
  // comment this out for real runs
  output.connect(Serial);
  dataStream.begin(9600); //Communication between devices, make this slower then the debug out
  skipToDelimiter = true; // skip first line, as may be part way through
}

void loop() {
  output.nextByteOut(); // send more buffered bytes
  if (input.readUntilToken(dataStream,dataLine,"\n",skipToDelimiter,false)) { // false => no echo back to stream
    if (parse(dataLine)) {
      showNewData();
    } else {
      // some data error
      output.print("data error:"); output.println(dataLine);
    }
  }
}

// returns false if data errors,
// i.e. not enough or too much data
// invalid int, missing field
bool parse(SafeString &data) {
  // look for , using SafeString stoken here see Serial Text I/O for the Real World examples
  // https://www.forward.com.au/pfod/ArduinoProgramming/Serial_IO/index.html
  size_t idx = 0;
  bool returnEmptyFields = true;
  cSF(sfField,19); // SafeString to hold field
  size_t i = 0;
  for (i = 0; i<(GRIDX * GRIDY); i++) {
    if (idx >= data.length()) { // test here so last i++ is executed
      break; // end of data
    }
    idx = data.stoken(sfField, idx, ",", returnEmptyFields); // 
   // output.print(i); output.print(" ");output.println(sfField);
    if (!sfField.toInt(myArray[i])) { // handles empty fields and non-ints
      // bad data
      output.print("bad data:");output.println(sfField);
      return false;
    }
  }
  if (i != (GRIDX * GRIDY)) {
    output.print("Error missing data only "); output.print(i+1); output.print(" found need "); output.println((GRIDX * GRIDY));
    return false; // missing data fields
  }
  if (idx != data.length()) {
     output.println("Too much data:");
     return false; // too much data
  }
  return true;
}


void showNewData() {
    output.println("Temperature array... ");
    for (int i = 0; i < GRIDY * GRIDX; i++) {
      output.print(myArray[i]);
      output.print(' ');
    }
    output.println();
}

Here is some sample test output

<Arduino is ready>
Temperature array... 
34 55 77 4 5 6 7 8 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 
bad data:a
data error:a, z, ,,10,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,36
Too much data:
data error:4,5,6,7,8,9,10,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,36,36,38