Go Down

Topic: analysing web client traffic (Read 194 times) previous topic - next topic

guy_c

Hello,
I was trying to visualize the traffic generated by the  web client in Examples > Ethernet
by David A. Mellis created 18 Dec 2009, modified 9 Apr 2012 by Tom Igoe, based on work by Adrian McEwen


The analyzer is a windows machine running a bridge and wireshark capturing traffic to/from arduino, please refer to attached figure

What i do not understand is all the packets (also to port 80) which always precede the http request (highlighted in yellow) coming from arduino.

These packets appear when the request is to google or to a wamp server on my LAN

when i replace arduino by a window machine making the same request with wget, the first packet is the always http request

I'd greatly appreciate if anyone could shade some light on this mystery

 


Peter_n

The WebClient tutorial : http://arduino.cc/en/Tutorial/WebClient

What is the contents of those packets ?
Here is explanation about the use of Wireshark with those packets : https://www.youtube.com/watch?v=GLw-qXdK1MM
Here is the same in the wiki : https://wiki.wireshark.org/TCP_Reassembly

I think this is about it: https://code.google.com/p/arduino/issues/detail?id=563#c7
The W5100 library perhaps does not buffer outgoing data, which result in so many TCP packets, and no one cares.

The W5100 is a TCP/IP chip. Perhaps the Ethernet library uses it as a simple TCP/IP chip without optimizations for HTTP requests.

SurferTim

#2
Mar 24, 2015, 01:06 pm Last Edit: Mar 24, 2015, 01:52 pm by SurferTim
Quote
The W5100 library perhaps does not buffer outgoing data, which result in so many TCP packets, and no one cares.
I care. I manage the packet size by the character array size I use to send packets with client.write. All my examples in the playground use character arrays to send bigger packets.

Some examples are not very internet friendly tho. They send a bunch of packets just to send one request. :(

edit: Here are two examples. The first sends 4 packets.
Code: [Select]
client.println("GET /mypage.php HTTP/1.1");
client.println("Host: mydomain.com");
client.println("Connection: close");
//...and the worst offender! A cr/lf sent in its own packet. :(
client.println();


This sends the same request in one packet.
Code: [Select]
client.write("GET /mypage.php HTTP/1.1\r\nHost: mydomain.com\r\nConnection: close\r\n\r\n");



guy_c

I understand: The request is segmented by arduino (here) because of the way the code is written. Wireshark (and of course the server) who's looking for \r\n\r\n knows when it's the last segment and shows the whole request on frame 19 though this frame has just the last \n byte. Indeed, when coded per SurferTim, the request is sent with just one 129 bytes frame.

Thanks Peter_n, Thanks SurferTim

Peter_n

SurferTim, can you confirm it is like this:

The total buffer inside the W5100 is 16kbyte, and it has 4 sockets, that is 2048 byte per buffer (a single TX or RX buffer for a socket) which is the number that is also used in the Ethernet library.
The data to the Ethernet is not written in a buffer, but directly passed on via SPI to the W5100 chip.

Therefor, the total length for a long text for a single Client.write can be up to 2048 (if the TX buffer is empty).
The free size could be checked with "W5100.getTXFreeSize(socket)", but with an Arduino Uno or Mega the 2048 bytes is probably never completely filled.

I still prefer seperate code lines in my sketch. I write it like this:
Code: [Select]

// optimize for single packet
client.print(F( "GET /mypage.php HTTP/1.1\r\n"
                "Host: mydomain.com\r\n"
                "Connection: close\r\n"
                "\r\n\r\n"));


Will the client.print() instead of client.write() still write the complete string ? I need the string to be in flash memory.

SurferTim

@Peter_n: That way is fine. zoomkat does the same thing with his client.print stuff.

AFAIK, the largest packet the w5100 can send is 1492 bytes. I don't think it can send anything larger than that. I could be wrong.

I use client.write because the call is faster, and I normally have the request formatted into a character array. If I need to build an array, I use strcpy_P and strcat_P and the PSTR macro. That keeps the static strings in program memory until I need them. Here is an example from my server code I have on the playground. The tBuf variable is a character array.
Code: [Select]
                // send Content-Type
                if(strcmp(fileType,"HTM") == 0) strcat_P(tBuf,PSTR("text/html"));
                else if(strcmp(fileType,"PHP") == 0) strcat_P(tBuf,PSTR("text/html"));
                else if(strcmp(fileType,"TXT") == 0) strcat_P(tBuf,PSTR("text/plain"));
                else if(strcmp(fileType,"CSS") == 0) strcat_P(tBuf,PSTR("text/css"));
                else if(strcmp(fileType,"GIF") == 0) strcat_P(tBuf,PSTR("image/gif"));
                else if(strcmp(fileType,"JPG") == 0) strcat_P(tBuf,PSTR("image/jpeg"));
                else if(strcmp(fileType,"JS") == 0) strcat_P(tBuf,PSTR("application/javascript"));
                else if(strcmp(fileType,"ICO") == 0) strcat_P(tBuf,PSTR("image/x-icon"));
                else if(strcmp(fileType,"PNG") == 0) strcat_P(tBuf,PSTR("image/png"));
                else if(strcmp(fileType,"PDF") == 0) strcat_P(tBuf,PSTR("application/pdf"));
                else if(strcmp(fileType,"ZIP") == 0) strcat_P(tBuf,PSTR("application/zip"));
                else strcat_P(tBuf,PSTR("text/plain"));

                strcat_P(tBuf,PSTR("\r\nConnection: close\r\n\r\n"));
                client.write(tBuf);

Peter_n


Go Up