Struggling with ESP8266 Client

[jstamp]

Hello All:
I'm trying to use the Sparkfun ESP8266 Shield to read data from a simple webpage. My long term goal is to be able to use php on a webserver to receive data from the Arduino (which will have sensors attached) and then the php on the server can send very simple pages back (for example, with the word "stop" which the arduino could detect with a fairly simple "if" statement) to perform some action.

I'm trying to work on baby steps, so my first goal was to read a simple webpage and print it to the serial monitor. No code to actually understand the web page is needed. I just want to read it and print it.

I'm mostly using the demo code from the shield with a few edits. I can't just use the demo code because it points to "example.com" and someone stopped paying the bill on that site

Here's some excerpts for what I think is causing the problem:

const char destServer[] = "sugarbushdata.com";


const String httpRequest = "GET /index.php HTTP/1.1\n"
                           "Host: sugarbushdata.com\n"
                           "Connection: close\n\n";

void clientDemo()
{
  // To use the ESP8266 as a TCP client, use the 
  // ESP8266Client class. First, create an object:
  ESP8266Client client;

  // ESP8266Client connect([server], [port]) is used to 
  // connect to a server (const char * or IPAddress) on
  // a specified port.
  // Returns: 1 on success, 2 on already connected,
  // negative on fail (-1=TIMEOUT, -3=FAIL).
  Serial.println(destServer);
  int retVal = client.connect(destServer, 80);
  if (retVal <= 0)
  {
    Serial.println(F("Failed to connect to server."));
   Serial.println(retVal);
    return;
  }


  // print and write can be used to send data to a connected
  // client connection.

  client.print(httpRequest);

  // available() will return the number of characters
  // currently in the receive buffer.
  while (client.available())
    Serial.write(client.read()); // read() gets the FIFO char
  
  // connected() is a boolean return value - 1 if the 
  // connection is active, 0 if it's closed.
  if (client.connected())
    client.stop(); // stop() closes a TCP connection.
}

What I get is a 400- your browser sent a request that this server could not understand.
It's printed out pretty nicely (ok, actually fairly ugly cause its the html code) on my serial monitor.

If I navigate to www.sugarbushdata.com I get my nice simple page.

(PS- I really want to read "index.php" off of the server because my eventual goal is to do the server-side processing with php. However, I stuck an index.html page up there just to make sure that wasn't the issue. I can't get either one of them to work.

I'm going to stick the whole code in the attatchments in case the problem is somewhere else in the file, but I'm pretty sure its an issue with what I'm sending the server...

THANK YOU in advance for your help. I've been tinkering with this for 2 days and I'm out of places to turn. Help me, Arduino.cc forum, you're my only hope

TestWifiConnectbroken.ino (6.62 KB)

[Juraj]

http requires \r\n as line end

[jstamp]

whoops- that would be a problem.

I updated the code to this:

const char destServer[] = "sugarbushdata.com";


const String httpRequest = "GET /index.php HTTP/1.1\r\n"
                           "Host: sugarbushdata.com\r\n"
                           "Connection: close\r\n";

My serial monitor gives me this:

ESP8266 Shield Present
Mode set to station
Connected to: MySpectrumWiFi84-2G
My IP: XXXXXXXXXXXX

Press any key to connect client.

sugarbushdata.com

Shouldn't it read out the contents of the webpage? I'm not seeing any response from the server. At least I'm not getting an error now...

[Juraj]

you don't wait for the response. it will not be immediately available

[jstamp]

Juraj:
you don't wait for the response. it will not be immediately available

Do I need a delay() line? Or is there a method that will let me wait until something is available...

[Juraj]

I usually use the blocking functions to read the response. they wait for a next character until timeout so they don't stop on gaps between packets and handle the initial gap too. this are readBytes, readBytesUntil, findUntil

I have a request to a server which takes really long to respond. I use

client.setTimeout(8000);
if (client.findUntil((char*) "HTTP/1.1 ", (char*) "\n")) {

findUntil waits for a character until timeout (and has wrong parameter types so the (char*) cast)

on other place I have simply
int readLen = modbus.readBytes(response, DATA_IX)

[jstamp]

Juraj:
I usually use the blocking functions to read the response. they wait for a next character until timeout so they don't stop on gaps between packets and handle the initial gap too. this are readBytes, readBytesUntil, findUntil

I have a request to a server which takes really long to respond. I use

client.setTimeout(8000);
if (client.findUntil((char*) "HTTP/1.1 ", (char*) "\n")) {

findUntil waits for a character until timeout (and has wrong parameter types so the (char*) cast)

on other place I have simply
int readLen = modbus.readBytes(response, DATA_IX)

So let me make sure I'm getting this straight:
.finduntil is a boolean- so when true it's waiting for data or has recieved data, false it timed out.

Serial.println(httpRequest); client.print(httpRequest);

client.setTimeout(8000);
if (client.findUntil((char*) "HTTP/1.1 ", (char*) "\n")) {

while (client.available())
Serial.write(client.read()); // read() gets the FIFO char
}
else {
Serial.println("Timeout");
}

Unfortunately, I'm timing out. I know there's a webpage there, because I can open it with Chrome. I'm now 90% sure I'm still screwing up the http request...

[Juraj]

add one more \r\n at the end of the request

[jstamp]

Juraj:
add one more \r\n at the end of the request

Still no luck. Here's the current request:

const char destServer[] = "sugarbushdata.com";


const String httpRequest = "GET /index.html HTTP/1.1\r\n"
                          "Host: sugarbushdata.com\r\n"
                           "Connection: close\r\n"
                           "\r\n";

Here's what I'm using to try and reach the server:

void clientDemo()
{
  // To use the ESP8266 as a TCP client, use the 
  // ESP8266Client class. First, create an object:
  ESP8266Client client;

  // ESP8266Client connect([server], [port]) is used to 
  // connect to a server (const char * or IPAddress) on
  // a specified port.
  // Returns: 1 on success, 2 on already connected,
  // negative on fail (-1=TIMEOUT, -3=FAIL).
  Serial.println(destServer);
  int retVal = client.connect(destServer, 80);
  if (retVal <= 0)
  {
    Serial.println(F("Failed to connect to server."));
   Serial.println(retVal);
    return;
  }


  // print and write can be used to send data to a connected
  // client connection.
Serial.println(httpRequest);
  client.print(httpRequest);

  client.setTimeout(8000);
    if (client.findUntil((char*) "HTTP/1.1 ", (char*) "\n")) {

  while (client.available())
    Serial.write(client.read()); // read() gets the FIFO char
    }
    else {
      Serial.println("Timeout");
    }
  // connected() is a boolean return value - 1 if the 
  // connection is active, 0 if it's closed.
  if (client.connected())
    client.stop(); // stop() closes a TCP connection.
}

IF I wrote that correctly, the if statement (client.findUntil) will be true as long as there's something there to find. It'll read "timeout" if it uses up the 8 seconds.

My serial monitor is displaying "timeout" BUT- it's not waiting 8 seconds. Did I screw up the code?

[Juraj]

you could try
while (!client.available());

to wait for response.

[jstamp]

Juraj:
you could try
while (!client.available());

to wait for response.

Thank you! That one got it. I stuck the while loop ahead of the write loop and it wrote out the text of html file.

I can't tell you how much I appreciate you sticking with me through this.

[marcvdm]

Hello everybody,

I'm POSTing data to a web server with a MKRNB1500 board, and that's working fine: the php script on the server receives this data in good order. Also, this script is printing some response text as output, that also works fine, since its visible when I browse to that page.
However, like in the previous posts of this thread, I'm trying to use:

while (client.available()>0) {
    Serial.print((char) client.read());
    delay(1);
  }

to catch the server response, but I don't get any available bytes ever, so there's never anything to client.read(). Even if I know the page output is there, for sure.

My question: do client.available() and client.read() work with any output printed by the server page? Do they handle the incoming response just like a bunch of characters? In that case, it should be ok to just echo a string of characters with php? Or do they first need proper headers, like HTTP 1.1, Content-length and so on?
If a specific format is required for the response to work with client.available() and client.read(), which one?

Beste regards, Marc

[Klaus_K]

@marcvdm Please do not hijack other posts. This will not help you. Old post with many answers get little attention. Create your own post. New post with good description and information tend to get more attention. You can link the old post into yours to let people know you read it already and suspect your issue is similar but still have questions.

I also recommend to read the "How to post". It is at the beginning of each sub forum. You used code tags already. That is a good sign. Post complete code.

Best way forward. Create a new post. Copy your stuff over and modify the post above, just in case one of the original poster reads this. You can leave a link to your own post.

Welcome to the forum