Creating a serial bridge from 1 esp32 board to another

This topic is specifically created to ask Cattledog a question / help because I was not able to send a message directly.

I have been doing extensive research and every time I find an answer in this or another forum, your name comes up among the responses that most often solve the problem (incl. a problem someone was having with esp32 freezing if the transmitted data was greater than 512 bytes), so naturally I assume that you must have extensive knowledge in the field of arduino / esp32 programming and bluetooth.

At first I tried to do this using BLE, but I quickly realized that it's very complicated plus it is designed for short bursts of transmission to save battery power, and although I got it to send data to some level, it turned out it would take too much idle time between transmissions so it wasn't a good solution. So I decided to use classic bluetooth / spp, which seems to be designed for this purpose.

I am trying to design / build a continuous serial data transmitter to link one rc radio transmitter to another transmitter. I have linked them using rx/tx and gnd serial wire, and it works great (it's one directional so only 1 tx wire and gnd is actually enough). The format is 100,000 baud, 8E2.

I am using "serial-bt".read() / write() and serial.read() write to accomplish this. (For testing purposes only) if I read the serial data on serial1 or the sending board and mirror it/write to the serial2 port of the same board, everything works fine. The read/write action keeps the data frame timed correctly, so it works ok.

The data is 25 bytes long every frame (12 bits per byte incl. the start bit, 8 data bits, 1 parity bit and 2 stop bits), so total of 300 bits at 100 kbauds and takes exactly 3 milliseconds to read/write. Each frame has an updated rate of 9 ms, so there is 6 ms of idle time (logic high for 6 ms).

After doing research and learning how to use the 64 bit timer of the esp32, I timed each task. The read on serial1 takes exactly 3ms, send to bt.write() takes another 1.5 ms, if I write to serial2 on the same board, it takes another 3 ms, and each "loop" cycle, depending on what all I do, takes about 9 to 10 ms.

My received signal on the receiving board was very erratic and did not reproduce the sent signal, after lots of reading I realized that there's no time reference on the 2nd board, so I have to use a timer to create the 9 ms update rate signal frame by use of timers, thd read/write tasks alone can't create a signal frame like on just one board. So I started timing the tasks on the receiving esp32 board. Everything seems reasonable, except the bt.read() task. For the 300 bits (25 bytes and writing them into an array buffer of [25] ) takes anywhere from 27 milliseconds to 42 milliseconds. It shouldn't take that long if bluetooth is capable of 1 Mbps or anything near that, but I don't think the transmission actually takes that long, it's the processing. I do all the reading/writing in the main loop. I tried just receiving and storing 1 byte of the 25, and it took 4 ms still, which seems long for just 1 byte. The remaining serial.write() and any other tasks take almost no time.
Any idea why receiving 300 bits (100 of which is stop bits and other overhead bits, so only 200 data bits) takes so long? When I try to reconstruct the signal frame with a refresh rate of 9 ms, if getting the new frame takes 27 to more than 40 ms, it would make for a very "slow" refresh rate. I know this can be done with classic bluetooth because some rc radios have internal bluetooth connection modules.

I use the regular server/client in the serial BT examples of the arduino IDE to establish a connection between the 2 esp32 boards, that part seems to work pretty well and easily. Any help would be appreciated to figure out how to speed up the reception part. If you know any additional info to see how I proceed, let me know. It's pretty simple so far though, just read 25 bytes from serial when available, send to bluetoothSerial, then the opposite on the receiving board, all in the main loop other than the initial setup.

Thank you

Just FYI, there are literally dozens of people here with the expertise to help you with your problem. But even Cattledog would probably need to see your code. An overview of your report suggests that you are overthinking what should be a very simple task.

To that end, it's super important to read the introductory information threads at the top of the forum. They outline the best way to present that information. It's simpler and less painful than having people guide and correct you about that, in a series of individual posts.

But I can give you a heads up, your complete sketch in code tags would be required to solve a problem like yours.

Yeah, no offense, but completely meaningless when seen out of context like this.

Sure, I'm fairly new here. I'll figure out how to post it in the correct format then post it here.

Thank you again

It's a good idea also, because your post count as a new user is limited on the first day. So you have a limited number of chances to post and reply, until you've been here longer...

Great thanks. I think there are better people than myself to address the code. But generally, from reading your description, you are depending on timing to frame the data. When working with serial, this approach is widely known to be a minefield full of problems, that frequently causes failures, requires overly complex code, and is extremely difficult to troubleshoot.

Usually, serial data is considered fully asynchronous, meaning there is no specific time frame for the arrival of characters. This is why traditionally, framing is always done with embedded symbols, or by decoding and recognizing encoded packets.

Thus, unless there is some reason why you can't send data defined packets or frames, vs. time defined packets or frames, the real answer to your problem is to eschew the time sensitive methods entirely.

For the 300 bits (25 bytes and writing them into an array buffer of [25] ) takes anywhere from 27 milliseconds to 42 milliseconds.

SerialBT.readBytes(byteSerial, 25);

I have no idea why the reading of 25 bytes should take the time you are measuring, and I have no direct experience with fast frame send/receive with BluetoothSerial.h

I wonder if with the 9 ms sending and the unsynchronized reading if there are buffer overflow issues. What happens if you send a single frame of 25 bytes. How long does the reading take?

As @ aarg has suggested, Serial communication without frame definition markers is difficult.

I'm open to define the frame using any required method. I'm not entirely sure what packed defined transmission would be. I assumed that's what the start and stop bytes are for, but not sure how you do it over bluetooth, since you can't define a BT baud rate or parity/stop bits. It actually took me a while to realize I needed to define the data frame by using timer registers.

I agree that serial is a fully asynchronous transmission method, however due to the nature of this being a control signal, frame rate and latency are important. It doesn't matter when the data arrives, and framing can be done by establishing a "zero" time basis when the first packet arrives, but since it is being sent with a set baud rate and refresh rate (based on the original input signal), it is important to reproduce that exact signal, whether it be every 9 ms or any other number of ms. However if it is done with a wired serial connection it can be done entirely asynchronous, the arriving data defines the frame and timing, so since BT spp is a direct replacement of wired serial connection, that same thing should be doable through BT classic. Since BT has no baud rate or stop/parity definition, then some timer framing seems to be required since those data markers are no longer available. For instance I can't define

SerialBT.begin(baudrate, 8E2, ... , ....) like I can with regular asynchronous serial.

I'm all for defining frame markers. It doesn't have to be without any markers, as long as I can reproduce the scope signal on the receiver end on the serial output. Even if it has to be done with raw reading of the flow/overhead bits then writing them as data bits on the output, it'll be fine, just have to reproduce the same 300 bits on the receiver serial TX pin same as transmitter RX pin.

I thought of BT rx buffer overflow when I read the problem you helped solve by increasing the input buffer from 512 bytes to 2048 byes, but I'm receiving only 25 bytes at a time, it's nowhere near overflowing the buffer.

As for receiving a single frame of 25 bytes, I'm not sure, I haven't done that, but with the current setup, even the 1st receive task takes about 30 ms. I even tried sending and receiving 1 byte at a time, but with a receive time of 3-4 ms for just 1 byte, that is not feasible because each 25 byte section has to be continuous for the signal to make sense to the rc radio.

Anyway, if this can be done in a different way, or with interrupts or inside functions instead to avoid problems, I'm open to other solutions. I'm just trying to make an esp32 to esp32 wireless transparent serial bridge. If SPP was initially created to replace serial communication without the wires, this should be doable by definition, right?

I'm not sure what bluetoothserial fast frame is as opposed to slow frame or any other option, I assumed this wasn't that fast compared to, for instance, transmission of audio data over a bluetooth emulated serial port to an audio device from a pc or phone, which requires a lot faster transmission rate than this, but if the transmitting esp32 board can send the entire 25 bytes in 2 milliseconds, it would make no sense that receiving the same number of bytes would take 15 to 20 times more time.

Thanks for your assistance.

Those pins can be problematic to use. Those pins must not be changed during booting or programming.

Interesting choices as baud rates.

The sender

#include <HardwareSerial.h>
HardwareSerial SerialTFMini( 1 );
HardwareSerial SerialBrain( 2 );

void setup() 
{

  Serial.begin( SerialDataBits );
  SerialBrain.begin( SerialDataBits );
  SerialTFMini.begin(  SerialDataBits, SERIAL_8N1, 27, 26 );
}

void fSendSerialToBrain( void *pvParameters )
{
  struct stu_LIDAR_INFO pxLIDAR_INFO;
  xSemaphoreGive ( sema_SendSerialToBrain );
  String sSerial;
  sSerial.reserve ( 300 );
  for (;;)
  {
    xEventGroupWaitBits (eg, evtSendSerialToBrain, pdTRUE, pdTRUE, portMAX_DELAY);
    // Serial.println ( " fSendSerialToBrain " );
    if ( xSemaphoreTake( sema_SendSerialToBrain, xZeroTicksToWait ) == pdTRUE ) // grab semaphore, no wait
    {
      int CellCount = 1;
      xSemaphoreTake( sema_LIDAR_INFO, xSemaphoreTicksToWait );
      xQueueReceive ( xQ_LIDAR_INFO, &pxLIDAR_INFO, QueueReceiveDelayTime );
      xSemaphoreGive( sema_LIDAR_INFO );
      sSerial.concat( "<!," ); //sentence begin
      sSerial.concat( String(ScanPoints) + "," );
      for ( CellCount; CellCount <= ScanPoints; CellCount++ )
      {
        sSerial.concat( String(pxLIDAR_INFO.Range[CellCount]) + "," );
      }
      sSerial.concat( ">" ); //sentence end
      // Serial.println ( sSerial );
      SerialBrain.println ( sSerial );
      //      ////
      //      xEventGroupSetBits( eg, evtSendLIDAR_Info_For_ALARM );
      //      xSemaphoreGive ( sema_SendSerialToBrain );
      sSerial = "";
    }
  }
  vTaskDelete( NULL );
} // void fSendSerialToBrain( void *pvParameters )

The receiver

#include <HardwareSerial.h>
// pin =RX, pin =TX
HardwareSerial GPSSerial ( 1 );
// pin 2=RX, pin 15=TX
HardwareSerial LIDARSerial ( 2 );
// pin 26=RX, pin 25=TX

void setup()
{
  LIDARSerial.begin ( SerialDataBits, SERIAL_8N1, 26, 25 );
  GPSSerial.begin ( GPS_DataBits, SERIAL_8N1, 2, 15 ); // begin GPS hardware serial
}

void fParseLIDAR_ReceivedSerial ( void * parameters )
{
  // distribute received LIDAR info
  String sTmp = "";
  sTmp.reserve ( 20 );
  String sMessage = "";
  sMessage.reserve ( StringBufferSize300 );
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtParseLIDAR_ReceivedSerial, pdTRUE, pdTRUE, portMAX_DELAY) ;
    xQueueReceive ( xQ_LIDAR_Display_INFO, &sMessage, QueueReceiveDelayTime );
    //        Serial.println ( sMessage );
    int commaIndex = sMessage.indexOf(',');
    sTmp.concat ( sMessage.substring(0, commaIndex) );
    sMessage.remove( 0, (commaIndex + 1) ); // chop off begining of message
    //    Serial.println ( sTmp );
    if ( sTmp == "!" )
    {
      xSemaphoreGive ( sema_LIDAR_OK );
      //  Display info from LIDAR
      sLIDAR_Display_Info = sMessage;
    }
    if ( sTmp == "$" )
    {
      xEventGroupSetBits( eg1, evtResetWatchDogVariables );
    }
    if ( sTmp == "#")
    {
      // Serial.println ( "#" );
      xSemaphoreTake( sema_LIDAR_Alarm, xSemaphoreTicksToWait );
      sLIDAR_Alarm_info = sMessage;
      xSemaphoreGive( sema_LIDAR_Alarm );
      xEventGroupSetBits( eg, evtfLIDAR_Alarm );
    }
    //    Serial.println ( "parse serial ok" );
    sTmp = "";
    sMessage = "";
    xSemaphoreGive( sema_ParseLIDAR_ReceivedSerial );
    //    Serial.print( "fParseReceivedSerial " );
    //    Serial.print(uxTaskGetStackHighWaterMark( NULL ));
    //    Serial.println();
    //    Serial.flush();
  }
  vTaskDelete( NULL );
} // void fParseReceivedSerial ( void * parameters )
//
void fReceiveSerial_LIDAR( void * parameters  )
{
  bool BeginSentence = false;
  sSerial.reserve ( StringBufferSize300 );
  char OneChar;
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtReceiveSerial_LIDAR, pdTRUE, pdTRUE, portMAX_DELAY);
    if ( LIDARSerial.available() >= 1 )
    {
      while ( LIDARSerial.available() )
      {
        OneChar = LIDARSerial.read();
        if ( BeginSentence )
        {
          if ( OneChar == ‘>’)
          {
            if ( xSemaphoreTake( sema_ParseLIDAR_ReceivedSerial, xSemaphoreTicksToWait10 ) == pdTRUE )
            {
              xQueueOverwrite( xQ_LIDAR_Display_INFO, ( void * ) &sSerial );
              xEventGroupSetBits( eg, evtParseLIDAR_ReceivedSerial );
              //
            }
            BeginSentence = false;
            break;
          }
          sSerial.concat ( OneChar );
        }
        else
        {
          if ( OneChar == ‘<’ )
          {
            sSerial = “”; // clear string buffer
            BeginSentence = true; // found begining of sentence
          }
        }
      } //  while ( LIDARSerial.available() )
    } //if ( LIDARSerial.available() >= 1 )
    xSemaphoreGive( sema_ReceiveSerial_LIDAR );
  }
  vTaskDelete( NULL );
} //void fReceiveSerial_LIDAR( void * parameters  )

Hope that tells a good story of doing serial ESP32 to ESP32.

The pins to use weren't my choice, since the io pins for Serial1 aren't available on the attached breakout board (gpio 10 & 11 I believe), I just picked a different pair of pins to use. I don't change them at any time, but if you think it makes a difference, I can use other pins. However, as mentioned earlier though, pins 12 & 13 are only to mirror the serial signal on the sending board, most of the time, after initial testing, they are disabled. They have nothing to do with the receiver board. On the receiving end, gpio 16 & 17 are used, which seem to be intrinsically tied to Serial2 anyway, so that shouldn't be causing a problem.

The baud rate isn't my choice. The rc radio output signal is defined/set at that exact baud rate, so I had to match the board settings to the exact setting of that signal. Believe me, if it were my choice, I would have picked something more trivial.

I am sure a baud of 115200 would not be pressed to do a slower speed.

I will try with 115200 where you have GPSDatabits and the other databits. Can I change the 8N1 to 8E2 if it doesn't work, or does it have to stay as 8N1?

The reason I went with 100 k baud rate was that someone told me if the original signal is 100 k, you can only weir off 2 or 3% before it starts causing problems, but if that's not correct, then 115k is fine with me.

Yes, in the sketch you posted with the slow BT receive time, the Serial issues do not appear to be relevant.

I don't see any reference to bluetooth.begin or anything like that. I know you are doing this much more efficiently than I did, but just a newb question just to make sure, do I still need to do the bluetooth initialization and connect the two boards as server and client as I did previously, or is that completely unnecessary and replaced by your code? I'm just wondering how the 2 boards will find each others RF / bluetooth signal.

You can change any setting you wish, it's your program. I've only posted a story of a way I have done ESP32 serial to serial communications. Interpret it as you wish. If you can use parts, great. if you get stuck, ask for help.

No. It's for one ESP32 to serially communicate with another ESP32 using wires. If you want to have ESP32's communicate wirelessly with each other there are several routes you can go. One way is by using ESPNOW.

In the receiver code of post #8, do you see the same 50 ms timing on the Bluetooth receive when there is no Serial output?