Go Down

Topic: Ethernet cleint println() makes new packet every time (Read 2 times) previous topic - next topic

mishoboss

Do you know that every time you use print() or println() there is a new TCP packet sent? For example look at this famous Pachube example: http://arduino.cc/en/Tutorial/PachubeCient

This is the code they use to send data to the pachube server:

Code: [Select]
client.print("PUT /api/YOUR_FEED_HERE.csv HTTP/1.1\n");
     client.print("Host: www.pachube.com\n");
     // fill in your Pachube API key here:
     client.print("X-PachubeApiKey: YOUR_KEY_HERE\n");
     client.print("Content-Length: ");

     // calculate the length of the sensor reading in bytes:
     int thisLength = getLength(thisData);
     client.println(thisLength, DEC);

     // last pieces of the HTTP PUT request:
     client.print("Content-Type: text/csv\n");
     client.println("Connection: close\n");

     // here's the actual content of the PUT request:
     client.println(thisData, DEC);


These are 7 separate packets.

I work on a project, that I need to combine 6 sensor's data (temperature, humidity and others) and send them to a web service every 2 seconds. However I want to do it in just ONE packet, not 14... There is no sense to send a TCP packet that could holds kilobytes with just one byte for example... Is there a way to concatenate strings, ints and floats in one string and send it via just one println()?

PaulS

Quote
Is there a way to concatenate strings, ints and floats in one string and send it via just one println()?

That's what the sprintf() function was invented for. Unfortunately, the Arduino implementation of sprintf doesn't handle floats. But, you could separate the float into its integer and fractional parts (scaled up), and output as two integers with a decimal point in between. Using the appropriate format specifier is essential.

mishoboss

Thanks for this, PaulS. However sprintf() uses a lot of RAM and my AT328 Arduino Uno gets easily restarted... I had this bad experience before too :(

PaulS

I guess you get to decide which short end of the stick you want to hold.

You don't necessarily need to use sprintf. You can emulate what it is doing with strcpy() and atoi() and an awareness of differing length strings (some of which need to be padded with 0s).

Jeffrey Sun

I have already created a Pachube library for Arduino. You can get it here (http://code.google.com/p/pachubelibrary/)

It supports the data types, such as String, int and float. Maybe it meets your requirement.

Getting into the source code of that library, you can find the function dtostrf can be used to convert the float data to string.

You are right. Every print(ln) function will send a TCP packet. But in your example, the 7 packets are finished in a TCP connection. And the web server can receive and parse the data correctly. Do you have any concern you want to send all data in a packet?

Of course you can put all the data to be sent into a string first, and then send the string by calling print once. But it will consume lots of continuous RAM when you populate the big string. I have tried this mean with 328p. The problem is I can't get such big continous RAM after running serveral hours.

Thanks,
Jeffrey

PaulS

Quote
The problem is I can't get such big continous RAM after running serveral hours.

The solution is to reuse the same RAM, statically allocated, over and over.

Jeffrey Sun

Right, that's a workaround. It works for the scenario that the max size of the data is known. But for the scenario the data size is dynamically changed on runtime, we need the allocate memory dynamically. I was also thinking if I can run a small memory management program to manage and clean memory fragment at runtime. But it seems unworthy to run it on such a small device.

robtillaart


The max TCP size packet is ~1500 bytes, you don't want a buffer that big.

Think the solution is in PaulS direction. Allocate a fixed byte array as send buffer (eg 80 bytes) and use dynamic memory as a 2nd order buffer, you could make a linked list of chuncks that are moved to the sendbuffer when it is emptied.



Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

mishoboss

Thanks, guys. I managed to put all http headers and data in a buffer of chars of just 187 bytes. Then I replace char by char only the data that is changed (a few chars) and send the whole buffer at once.

I have another question. Which is better, to make a new connection every time I want to transmit and then close it. Or connect once in the setup() function, continuously checking in the loop() if we are connected (if not call client.stop() then client.connect() to connect again) and call client.println() whenever I like?

If we use the first method, every time we transmit there is a new TCP connection created with a new port. If we use the second method, we use that connection for many transmits, saving some handshaking and routing... I think.

Which method is better?

robtillaart

Quote
Which method is better?


Depends on the amount of data send. (rule of thumb)

Reconnecting takes more time, but you can do better debugging, and you have resources free for other connections (optionally)
The latter is also true for the server at the other side !!!

However if you need to send much data you better keep the socket open, until done

my 2 cnts
Rob
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

mishoboss

After some tests I end up with a new connection on every transmit. It's more reliable I think.

When I tested with one connection for many transmits, the connection hangs after some minutes and takes a while to reconnect, loosing some packets meanwhile. Maybe this is due to a bad logic and code, I don't know.

The data I transmit is short (187 bytes), but I send it 3 times every 2 seconds. Every time I send data, connect with a new port. I'm not sure this is the best approach, but works perfectly so far.

Another question - after disconnecting the cable from the RJ-45 of the Arduino Shield, the Arduino hangs. I tried everything, but still the same. Is it something with the Ethernet library, or it's something in my code?

robtillaart

Quote
Is it something with the Ethernet library, or it's something in my code?


Not heard such behavior of the Ethernet-lib.
If you want to you can post your code so the forum can review it ..
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

mishoboss

#12
Jul 21, 2011, 10:57 am Last Edit: Jul 21, 2011, 02:12 pm by mishoboss Reason: 1
I attach the whole .pde as it exceeds the maximum allowed post length.
This is 5-zone thermostat that is controllable via REST. It sends data via REST too.

P.S. I localized the problem into the sending scope. It is somewhere here:

Code: [Select]
void sendRESTdata(int n, int c){

     if (clientREST.connect()) {
     

     if(c == 1){
        buffer[22] = 't'; // send temperature
        dtostrf(temperatures[n], 3, 1, temp);
        k=0;
        while(temp[k] != '\0'){
            buffer[k+172] = temp[k];
            k++;
        }
        buffer[k+172] = '\0'; 


        buffer[107] = char(48+k);
     } else if(c == 2){ // send humidity
        buffer[22] = 'h';
        dtostrf(humidities[n], 3, 1, temp);
        k=0;
        while(temp[k] != '\0'){
            buffer[k+172] = temp[k];
            k++;
        }   
        buffer[k+172] = '\0';
        buffer[107] = char(48+k);
     } else if(c == 3){ // send setpoint
        buffer[22] = 's';
       
        buffer[172] = char(48+setPoints[n]) ;
        buffer[173] = '\0';
        buffer[107] = char(49);
     } else if(c == 4){ // send isHeating
        buffer[22] = 'i';
       
        buffer[172] = char(48+isHeating[n]); 
        buffer[173] = '\0';
        buffer[107] = char(49);
     }
     
     buffer[20] = char(48+n);
     clientREST.println(buffer);
     clientREST.stop();
}
   

}


Arduino freezes only when the cable is unplugged. If I try to connect to a server that doesn't exists, there's no problem.

mishoboss

Quote
Arduino freezes only when the cable is unplugged. If I try to connect to a server that doesn't exists, there's no problem.

Well, it turned out this is not entirely right. It happens not only when the cable is unplugged, but also when the target server (as a machine) is shut down. I try to connect from the Arduino to a server listening on port 8080. If this server is not running, but the PC is running, there's no problem with the Arduino. However if the PC that runs this server is shut down, the Arduino freezes for a couple of seconds every time it tries to make a connection to the PC.

So, the problem is definitely the software. Any ideas? :(

Adrian McEwen

I'm not sure what you mean by "freezes"?  If the server isn't turned on then there won't be any reply to the packets the Arduino sends out to try to bring up the connection.  The Arduino will spend some time (it's usually ~40 seconds on most TCP implementations but I've not double-checked the Arduino one) retrying the connection before it gives up.

So in your code it would spend a while inside the clientREST.connect() call if the server isn't on at all.  For the case when the PC is turned on, but the server program isn't running, the PC will respond to the first connection request with a reset packet, and so the Arduino will give up trying to connect almost immediately.

Hope that helps,

Adrian.

Go Up