Ethernet cleint println() makes new packet every time

Using Arduino Networking, Protocols, and Devices
1

Do you know that every time you use print() or println() there is a new TCP packet sent? For example look at this famous Pachube example: http://arduino.cc/en/Tutorial/PachubeCient

This is the code they use to send data to the pachube server:

client.print("PUT /api/YOUR_FEED_HERE.csv HTTP/1.1\n");
     client.print("Host: www.pachube.com\n");
     // fill in your Pachube API key here:
     client.print("X-PachubeApiKey: YOUR_KEY_HERE\n");
     client.print("Content-Length: ");

     // calculate the length of the sensor reading in bytes:
     int thisLength = getLength(thisData);
     client.println(thisLength, DEC);

     // last pieces of the HTTP PUT request:
     client.print("Content-Type: text/csv\n");
     client.println("Connection: close\n");

     // here's the actual content of the PUT request:
     client.println(thisData, DEC);

These are 7 separate packets.

I work on a project, that I need to combine 6 sensor's data (temperature, humidity and others) and send them to a web service every 2 seconds. However I want to do it in just ONE packet, not 14... There is no sense to send a TCP packet that could holds kilobytes with just one byte for example... Is there a way to concatenate strings, ints and floats in one string and send it via just one println()?

2

Is there a way to concatenate strings, ints and floats in one string and send it via just one println()?

That's what the sprintf() function was invented for. Unfortunately, the Arduino implementation of sprintf doesn't handle floats. But, you could separate the float into its integer and fractional parts (scaled up), and output as two integers with a decimal point in between. Using the appropriate format specifier is essential.

3

Thanks for this, PaulS. However sprintf() uses a lot of RAM and my AT328 Arduino Uno gets easily restarted... I had this bad experience before too :frowning:

4

I guess you get to decide which short end of the stick you want to hold.

You don't necessarily need to use sprintf. You can emulate what it is doing with strcpy() and atoi() and an awareness of differing length strings (some of which need to be padded with 0s).

5

I have already created a Pachube library for Arduino. You can get it here (http://code.google.com/p/pachubelibrary/)

It supports the data types, such as String, int and float. Maybe it meets your requirement.

Getting into the source code of that library, you can find the function dtostrf can be used to convert the float data to string.

You are right. Every print(ln) function will send a TCP packet. But in your example, the 7 packets are finished in a TCP connection. And the web server can receive and parse the data correctly. Do you have any concern you want to send all data in a packet?

Of course you can put all the data to be sent into a string first, and then send the string by calling print once. But it will consume lots of continuous RAM when you populate the big string. I have tried this mean with 328p. The problem is I can't get such big continous RAM after running serveral hours.

Thanks,
Jeffrey

6

The problem is I can't get such big continous RAM after running serveral hours.

The solution is to reuse the same RAM, statically allocated, over and over.

7

Right, that's a workaround. It works for the scenario that the max size of the data is known. But for the scenario the data size is dynamically changed on runtime, we need the allocate memory dynamically. I was also thinking if I can run a small memory management program to manage and clean memory fragment at runtime. But it seems unworthy to run it on such a small device.

8

The max TCP size packet is ~1500 bytes, you don't want a buffer that big.

Think the solution is in PaulS direction. Allocate a fixed byte array as send buffer (eg 80 bytes) and use dynamic memory as a 2nd order buffer, you could make a linked list of chuncks that are moved to the sendbuffer when it is emptied.

9

Thanks, guys. I managed to put all http headers and data in a buffer of chars of just 187 bytes. Then I replace char by char only the data that is changed (a few chars) and send the whole buffer at once.

I have another question. Which is better, to make a new connection every time I want to transmit and then close it. Or connect once in the setup() function, continuously checking in the loop() if we are connected (if not call client.stop() then client.connect() to connect again) and call client.println() whenever I like?

If we use the first method, every time we transmit there is a new TCP connection created with a new port. If we use the second method, we use that connection for many transmits, saving some handshaking and routing... I think.

Which method is better?

10

Which method is better?

Depends on the amount of data send. (rule of thumb)

Reconnecting takes more time, but you can do better debugging, and you have resources free for other connections (optionally)
The latter is also true for the server at the other side !!!

However if you need to send much data you better keep the socket open, until done

my 2 cnts
Rob

11

After some tests I end up with a new connection on every transmit. It's more reliable I think.

When I tested with one connection for many transmits, the connection hangs after some minutes and takes a while to reconnect, loosing some packets meanwhile. Maybe this is due to a bad logic and code, I don't know.

The data I transmit is short (187 bytes), but I send it 3 times every 2 seconds. Every time I send data, connect with a new port. I'm not sure this is the best approach, but works perfectly so far.

Another question - after disconnecting the cable from the RJ-45 of the Arduino Shield, the Arduino hangs. I tried everything, but still the same. Is it something with the Ethernet library, or it's something in my code?

12

Is it something with the Ethernet library, or it's something in my code?

Not heard such behavior of the Ethernet-lib.
If you want to you can post your code so the forum can review it ..

13

I attach the whole .pde as it exceeds the maximum allowed post length.
This is 5-zone thermostat that is controllable via REST. It sends data via REST too.

P.S. I localized the problem into the sending scope. It is somewhere here:

void sendRESTdata(int n, int c){

     if (clientREST.connect()) {
      

     if(c == 1){
        buffer[22] = 't'; // send temperature
        dtostrf(temperatures[n], 3, 1, temp);
        k=0;
        while(temp[k] != '\0'){
            buffer[k+172] = temp[k];
            k++;
        }
        buffer[k+172] = '\0';  

 
        buffer[107] = char(48+k);
     } else if(c == 2){ // send humidity
        buffer[22] = 'h';
        dtostrf(humidities[n], 3, 1, temp);
        k=0;
        while(temp[k] != '\0'){
            buffer[k+172] = temp[k];
            k++;
        }   
        buffer[k+172] = '\0'; 
        buffer[107] = char(48+k);
     } else if(c == 3){ // send setpoint 
        buffer[22] = 's';
        
        buffer[172] = char(48+setPoints[n]) ; 
        buffer[173] = '\0'; 
        buffer[107] = char(49);
     } else if(c == 4){ // send isHeating 
        buffer[22] = 'i';
        
        buffer[172] = char(48+isHeating[n]);  
        buffer[173] = '\0'; 
        buffer[107] = char(49);
     }
     
     buffer[20] = char(48+n);
     clientREST.println(buffer);
     clientREST.stop();
}
    

}

Arduino freezes only when the cable is unplugged. If I try to connect to a server that doesn't exists, there's no problem.

thermostatHA_REST.pde (9.87 KB)

14

Arduino freezes only when the cable is unplugged. If I try to connect to a server that doesn't exists, there's no problem.

Well, it turned out this is not entirely right. It happens not only when the cable is unplugged, but also when the target server (as a machine) is shut down. I try to connect from the Arduino to a server listening on port 8080. If this server is not running, but the PC is running, there's no problem with the Arduino. However if the PC that runs this server is shut down, the Arduino freezes for a couple of seconds every time it tries to make a connection to the PC.

So, the problem is definitely the software. Any ideas? :frowning:

15

I'm not sure what you mean by "freezes"? If the server isn't turned on then there won't be any reply to the packets the Arduino sends out to try to bring up the connection. The Arduino will spend some time (it's usually ~40 seconds on most TCP implementations but I've not double-checked the Arduino one) retrying the connection before it gives up.

So in your code it would spend a while inside the clientREST.connect() call if the server isn't on at all. For the case when the PC is turned on, but the server program isn't running, the PC will respond to the first connection request with a reset packet, and so the Arduino will give up trying to connect almost immediately.

Hope that helps,

Adrian.

16

Thank you. It helps a lot! :slight_smile:
So, the next question is how could I overcome this ~40 seconds try? Or at least how can I make it not occupy all Arduino resources? I have a program on the Arduino that needs to run smooth, no matter if there's connection to the server.

17

Anyone any idea or hint?

18

Or at least how can I make it not occupy all Arduino resources?

The wording of this question implies a lack of understanding of the Arduino. The question implies that the thread that is waiting for a connection is block all other threads. Well, that is both true and false. It is false in the sense that there are no other threads to block. It is true in the sense that the connect function is indeed blocking the (only) thread.

Changing the connect function will require a thorough understanding of all of the code involved in creating (and waiting for) the connection. You will most likely need to change the number of times the code tries to connect, and the amount of time it waits for a connection.

Either of these changes will reduce the likelihood of a successful connection. So, you'll need to compensate for this, somehow.

19

Well, I know pretty well Arduino isn't C#, Java or similar managed code language :wink:
However, I thought there is a nice solution about this. Arduino is very popular platform to connect to pachube and other services and this "freezing" if there's no server is no good.

20

So, the next question is how could I overcome this ~40 seconds try? Or at least how can I make it not occupy all Arduino resources?

Have you looked in the ethernet library to see if there are some default time settings that can be changed?