Help with reading serial data from ESP8266

Hello all!

I'm trying to interface an ESP8266 with my Tensy 3.1 board without use of a third party library. I have a Java server set up that the ESP8266 can communicate with. I have communicated with this Java server successfully using a serial-passthrough approach. All the server does currently is respond to anything with 101 characters of numbers and then a squiggle (~). However, I am failing to receive the desired response from the ESP8266 when using a function I am trying to write to read the data in from the ESP8266.

The function in question is named "readBuffer". The goal of this function is to:

  1. Wait for data to become available from the ESP8266 (Serial1), or time out if it takes too long for data to become available

  2. Read this data in

  3. Return a String object with the characters read in from the ESP8266

Essentially, I should be able to shoot the ESP8266 a command and then call my readBuffer function after that to read in all relevant response data. Below is my code.

int led = 13;

void setup() {   
  pinMode(led, OUTPUT);             
  Serial.begin(9600);
  Serial1.begin(9600);
  blinkLED(500);
  blinkLED(500);
  blinkLED(500);
  Serial.println("Serial ready!");   
}

void loop() {
  blinkLED(100);
  if (Serial.available()) // Inside this if statement is just stuff to send data from computer Serial monitor to ESP8266
  {
  while(Serial.available())
  {
    Serial1.print((char)Serial.read());
  }
  Serial1.print("\r\n");
  }
  //If this commented out code below is uncommented and used instead of the actual code below this, the read appears correctly
/*
  while(Serial1.available())
  {
    Serial.print((char)Serial1.read());
    delay(1);
  }
  */
  
 String printOut = readBuffer(1000);
 if(!printOut.equals("TIMEOUT"))
 Serial.println(printOut);
 
  
}

void blinkLED(int delayTime)
{
  digitalWrite(led, HIGH);  
  delay(delayTime/2);               
  digitalWrite(led, LOW);   
  delay(delayTime/2);               
}

/*
 * ESP8266 Functions
 */

String readBuffer(int waitTimeout)
{
  int iterator; // External iterator variable for the second for loop
  char received[512]; // A place to throw received data
  for(int i = 0; ((i < waitTimeout) && (!Serial1.available())); i++) // This loops is here to wait for Serial1 to become available
  {
    delay(1);
  }
  if (!Serial1.available()) // This will fire if the for loop ended because the timeout was exceeded
  {
    return "TIMEOUT";
  }
  for(iterator = 0; (iterator < 511) && (Serial1.available()); iterator++){ // Here's the juicy stuff; while Serial1 is available and fewer than 511 characters have been read in, continue to read characters in
    received[iterator] = (char)Serial1.read(); //Read in one more character
    delay(1); // Wait a little
    for(int j = 0; ((j < 100) && (!Serial1.available())); j++) //Wait up to 100ms for the next character to become available if the above 1ms wasn't enough
    {
      delay(1);
    }
  }
  received[iterator] = '\0'; // close our "string" before returning it as an actual String
  return String(received);
}

 /*
  * End ESP8266 Functions
  */

Okay. Cool. So, what problems am I having with it? Well, if I use the code exactly as is and run the following commands on the ESP8266, it looks like:

Serial ready!
AT+CIPSTART="TCP","192.168.1.122",2468


OK
Linked

AT+CIPSEND=7

> 
hello

SEND OK

+IPD,202:

OK
Unlink

And here (below) is what it is supposed to look like.

Serial ready!
AT+CIPSTART="TCP","192.168.1.122",2468


OK
Linked

AT+CIPSEND=7

> 
hello

SEND OK

+IPD,202:0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789~

OK
Unlink

I see this response in the serial monitor when I don't use my function and just use the commented out while loop in my code (inside of the loop() method). Another interesting thing I ran into while writing this post is that I cannot copy and paste that correct response from serial monitor into notepad. It shows up blank in notepad after "+IPD,202:" even though I clearly can see it in serial monitor. What the heck is going on? Anybody have any ideas? Am I missing something silly?

Sorry for the lengthy post. Wanted to give as much necessary contextual information as I could.

I figured I might as well include my Java server code as well just in case one of you is willing to dig through it as well. It uses the Netty 4 library, which I am new to, but seems to be a very simple server to me. It just replies to anything with the same response. I also have a simple client in the attached zip to test the server out with.

javaServerStuff.zip (2.7 KB)

The examples in Serial Input Basics may be useful.

Your while(Serial.available()) is not a robust way to receive data as you have no means to know if all the data has arrived.

...R

Thank your for your reply. However, I was not able to pull anything new or applicable from my quick run through your post. I understand that my code is not "fast" in that it is blocking and waits when it may not have to, but my issue is not with the speed of my code. Also, my code does wait for the end of a response. See the loop by the comment " //Wait up to 100ms for the next character to become available if the above 1ms wasn't enough". This loop should allow up to 100ms to pass between bytes received. If no next byte is received after a whole tenth of a second, then the transmission is considered finished.

My issue is with the fact that I can correctly see the output from the ESP8266's messages (things like "OK" and output from "AT+CWLAP"), but I cannot correctly see output from anything after the "+IPD," marker besides the "OK".

Is this a character encoding problem, perhaps? Why can Serial.print(char) show the data (but then that can't be copied into notepad), but when the data is shoved into a String and then printed, it is blank? (Please excuse the awkward wording of that sentence)

I don't have time to test this theory in depth now, but after digging through the Netty documentation and some Java documentation it became clear that "char" in java is UTF-16 (two bytes), while in C it is 1 byte (UTF-8?). I'll play with this more when I get time. I have a strong feeling this may be causing the issue.

I can confirm now that the issue was the fact that Java chars are 2 bytes and C chars are 1 byte. Instead of using the char datatype on the Java side, I switched to the byte datatype (cast char to byte) and things seem to be running correctly now.

Glad you have found the problem

sopwyin54:
This loop should allow up to 100ms to pass between bytes received. If no next byte is received after a whole tenth of a second, then the transmission is considered finished.

I don't like relying on "wasteful" time intervals when receiving data, but I can see how it can work if the Arduino has nothing better to do.

...R