Reading HTTP response with SIM900 chip

I’m using a SIM900 chip to communicate with a backend over HTTP.

Everything is working fine in the sense that I can initiate a TCP connection to the server and execute an HTTP POST using the standard modem AT commands like this :

AT+CIPSEND=185
AT+CIPSTART="TCP","backendserver.com","80"
AT+CIPSEND=185
>

POST /endpointurl
HOST: backendserver.com
User-Agent: Arduino

The request reaches the server fine and the response is also received by the SIM900 chip.

However, when I try to read the response using the following code :

  int i = 0;
  while(_serial->available() && i<999 ) { 
        Serial.print((char)_serial->read());
    i++;
  }

I’m only getting a small portion of the response :

HTTP/1.1 500 Internal Server Error
Date: Mon, 16 Dec 2013 15

It stops at around 60 characters into the response

When I execute the POST with curl I see the complete response (HTTP headers + content) :

HTTP/1.1 500 Internal Server Error
Date: Mon, 16 Dec 2013 15:07:39 GMT
Server: Apache-Coyote/1.1
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked
< 
* Closing connection #0
{"errorMsg":"Error occured during update","errorCode":-1}

Any idea why I’m only seeing a small part of the response when reading from the serial line ?

Why, when the server tells you that what you are trying to do failed, does it matter?

That's not the point ... I could be interested in the errorCode for one thing. The fact that this returns an HTTP 500 doesn't matter. I have the same issue with an HTTP 200.

the reason it seems to 'hack off' the response after around 60 characters is because the Arduino's Serial buffer is full. What you can do is inserting a delay(1000) or so after each 60 chars.

Use slightly smarter read loop. As pointed out, you are reading the input buffer faster than it gets filled.

I'm not behind my pc now, but i use a read loop that only aborts when no data is received within say 10msecs. When a byte is received within that time, the counter is reset. My routine also aborts when '\r\n' is received.so basically it is a readLine with a tmeout.

patrick_reynen: the reason it seems to 'hack off' the response after around 60 characters is because the Arduino's Serial buffer is full. What you can do is inserting a delay(1000) or so after each 60 chars.

How will more delay() stop the serial buffer from overflowing? Less delay() makes sense. More does not.

PaulS:

patrick_reynen: ....

How will more delay() stop the serial buffer from overflowing? Less delay() makes sense. More does not.

The while loop has no delay inside the loop, therefore it is reading bytes from the buffer as fast as possible. So fast in fact, that it reads bytes faster than they arrive (this is a hardware serial, so it happens at the same time). The buffer you are reading will at some point be fully read, while the bytes are still coming in: the 'available()’ will return 0 and your read loop quits before the entire datastream has been read.

A less optimal (but working) loop will be something like

while(Serial.available() > 0) {
   c = Serial.read();
   delay(5);     //give the sender a chance to fill another character in the read buffer
};

This can of course be hugely optimized, but i just mention it as an example.

The while loop has no delay inside the loop, therefore it is reading bytes from the buffer as fast as possible. So fast in fact, that it reads bytes faster than they arrive (this is a hardware serial, so it happens at the same time). The buffer you are reading will at some point be fully read, while the bytes are still coming in

My first thought was "Hogwash". But, I'll skip voicing that thought.

The correct procedure IS to read serial data as fast as possible BUT NOT TAKE ACTION until the end of packet marker arrives. The HTTP response DOES have a way of indicating the size of the packet or that this is the end of the packet. It is up to the device user to determine what that end of packet marker is, or the size of the packet, so that you know when the end of the packet arrives.

Loitering in the hopes that the end of the packet will arrive before you get done reading is a fool's "solution".

Like i said, this is to explain the basic direction, not a correct solution. Its only intended to demonstrate the buffer effect.

That said, my own readLine routine reads as aggressively as possible until \r\n (packet end as you call it), but when no data is present it waits until a timeout occurs so that i know i’m getting bad responses.

Ps. Being dutch, I have no idea what hogwash is. I just try to help.