Arduino HTTP server sending malformed packets

Using Arduino Networking, Protocols, and Devices
#1

I’ve been having some issues with my Arduino sketch (lock ups/slow response). I have fixed the lock up issue with some code cleanup, but the web page the Arduino serves up is still slower than I would expect. I decided to sniff the IP traffic between the Arduino and my network and was surprised to see a bunch of “malformed” packets. I’ve attached my sketch here for reference, but I get these same malformed packets even when running a stripped down, simplified version of the sketch. I’ve also attached a pcap file containing the traffic coming from the Arduino (192.168.10.102) to the client (192.168.10.103).

The webpage loads, so these malformed packets aren’t breaking anything, but I think they may have to do with the slowness.

Any ideas?

http.pcap (46.3 KB)

Alarm_dev.ino (8.58 KB)

#2

To rule out any problems in my code, I tried running the example server sketch (http://arduino.cc/en/Tutorial/WebServer). This also produces what Wireshark labels as malformed packets. I've attached a packet capture of this example sketch running on official Arduino Uno R3 with an official Arduino Ethernet shield R3. What's going on here?

#3

rtadams89: I've been having some issues with my Arduino sketch (lock ups/slow response). I have fixed the lock up issue with some code cleanup, but the web page the Arduino serves up is still slower than I would expect. I decided to sniff the IP traffic between the Arduino and my network and was surprised to see a bunch of "malformed" packets. I've attached my sketch here for reference, [u]but I get these same malformed packets even when running a stripped down, simplified version of the sketch[/u]. I've also attached a pcap file containing the traffic coming from the Arduino (192.168.10.102) to the client (192.168.10.103).

The webpage loads, so these malformed packets aren't breaking anything, but I think they may have to do with the slowness.

Any ideas?

Malformed packets are COMPLETELY NORMAL, it is when wireshark does not have all the information to parse the full stream.

HOWEVER in your case Use your wireshark,->analyze-> follow stream-> hex dump

00000000  47 45 54 20 2f 20 48 54  54 50 2f 31 2e 31 0d 0a GET / HT TP/1.1..
00000010  41 63 63 65 70 74 3a 20  74 65 78 74 2f 68 74 6d Accept:  text/htm
00000020  6c 2c 20 61 70 70 6c 69  63 61 74 69 6f 6e 2f 78 l, appli cation/x
00000030  68 74 6d 6c 2b 78 6d 6c  2c 20 2a 2f 2a 0d 0a 41 html+xml , */*..A
00000040  63 63 65 70 74 2d 4c 61  6e 67 75 61 67 65 3a 20 ccept-La nguage: 
00000050  65 6e 2d 75 73 0d 0a 55  73 65 72 2d 41 67 65 6e en-us..U ser-Agen
00000060  74 3a 20 4d 6f 7a 69 6c  6c 61 2f 35 2e 30 20 28 t: Mozil la/5.0 (
00000070  63 6f 6d 70 61 74 69 62  6c 65 3b 20 4d 53 49 45 compatib le; MSIE
00000080  20 39 2e 30 3b 20 57 69  6e 64 6f 77 73 20 4e 54  9.0; Wi ndows NT
00000090  20 36 2e 31 3b 20 57 4f  57 36 34 3b 20 54 72 69  6.1; WO W64; Tri
000000A0  64 65 6e 74 2f 35 2e 30  29 0d 0a 41 63 63 65 70 dent/5.0 )..Accep
000000B0  74 2d 45 6e 63 6f 64 69  6e 67 3a 20 67 7a 69 70 t-Encodi ng: gzip
000000C0  2c 20 64 65 66 6c 61 74  65 0d 0a 48 6f 73 74 3a , deflat e..Host:
000000D0  20 31 39 32 2e 31 36 38  2e 31 30 2e 31 30 32 0d  192.168 .10.102.
000000E0  0a 43 6f 6e 6e 65 63 74  69 6f 6e 3a 20 4b 65 65 .Connect ion: Kee
000000F0  70 2d 41 6c 69 76 65 0d  0a 0d 0a                p-Alive. ...
    00000000  48                                               H
    00000001  54                                               T
    00000002  54                                               T
    00000003  50                                               P
    00000004  2f                                               /
    00000005  31                                               1
    00000006  2e                                               .
    00000007  31                                               1
    00000008  20                                                
    00000009  32                                               2
    0000000A  30                                               0
    0000000B  30                                               0
    0000000C  20                                                
    0000000D  4f                                               O
    0000000E  4b                                               K
    0000000F  0a                                               .

now go back to your packet window look at the first 'malformed packet' packet by selecting it and in you bottom window wire shark will have a list say:

Frame 5:60 bytes on wire ethernet internet protocol transmission control ............... SEQ:1, ACK:252 , Len:1 HYPER TRANSFER PROTOCOL

now select "HYPER TRANSFER PROTOCOL" and look in the HEX dump at WHICH character is highlighted: in this case it is a "H", now match it up with the hex dump above

move down 1 packet to frame 6 and it is a "T" packet 7 is your client saying thankyou 8 & 9 is "T","P" and so on.. so now you know that the "malformed packets " are not a 'problem' in so far as the content is correct, just some nutter(you ;)) is forcing the system to send HTTP requests one character at a time.....

wireshark is useful, but you must know how to interpret the output correctly.

Now all you need to do is re-jig your code and it will be faster than shit down a vertical slope.

HC

#4

I'm bookmarking this page, for reference any time I have communications issues!

#5

I realized that it was ending data in a very inefficent way (I'm trying to get client.write() to work as I understand it will not do this), but I didn't realize Wireshark would see these as "malformed".

#6

Technically speaking they are 'de-foremed' in the context of a packet being complete HTTP, wait until you see packets for warning full buffers and empty buffers in your Ethernet card. There has been a real fight on the wire-shark forums over how these packets should be displayed.

Next part is tracking this down, if I was being vindictive I would say the fault is in the Ethernet 'printlin' library.

Why don't you try the "tinywebserver" code, so that you can concentrate of the interface and not worry about dealing with HTTP at this level, it would also mean that fixes don't require plowing through endless println code.

#7

PaulS: I'm bookmarking this page, for reference any time I have communications issues!

Absolutely! Plus the stuff about TCP was interesting too. :)

#8

I can solve the problem by using client.write() in place of client.print(). However, that also means I can’t just wrap strings in F() to store them in flash instead of RAM. Is there a easy way to convert a bunch of lines like

client.print(F("<form method='GET'>\n<input name='random' type='hidden' value='"));

to use write(), but still store the text string in flash instead of RAM? Maybe a function that loops through the F(String) passed to it and returns a char array?

#9

Further testing has shown that using client.write() helps, but I am unable to impliment this in my sketch without using up too much RAM. I can't figure out a way to continue to store the text strings in flash, and allow them to be sent with write(). Any help?

#10

Have you ever wondered why there is a microSD card reader on most ethernet shields? XD

#11

Sure, that is a possibility, but even if I store the text on my SD card (which seems like a waste of a SD card, when I have program space to store the text in), I haven’t been able to successfully send it to the client with write(), only with print().

#12

Taken from an Adventure game I was working on:

/* Print a string from Program Memory directly to save RAM */
void printProgStr (const char * str)
{
  char c;
  if(!str) return;
  while((c = pgm_read_byte(str++)))
    lcd.print(c);
} // end of printProgStr

You pass a PROGMEM string to it, and it pulls it out a byte at a time and does lcd.print on each one. Just change the lcd.print to client.write() and you should be there. There’s probably a library function that does it, but I am too lazy to look it up. :wink:

#13

Won’t doing it that way negate the benefit of using client.write() (which is that it sends multiple bytes per packet), since it is passing a single byte to the .write() function each time.

#14

Then you should allocate a buffer (64 bytes or more), get a buffer full of data, then send it all at once.

#15

This issue I run into is that I cannot convert the string (which is of type: const __FlashStringHelper*) into a char array. If I could do that, eitherer with a built in or custom function, I could simply pass that char array to client.write().

#16

Here:

/* Print a string from Program Memory directly to save RAM */
void printProgStr (__FlashStringHelper * str)
{
  char * p = (char *) str;
  if (!p) return;  // no string, uh oh!

  char buf [strlen_P (p) + 1];  // dynamic allocation ;)
  byte i = 0;
  char c;
  while ((c = pgm_read_byte(p++)))
    buf [i++] = c;
  buf [i] = 0;
  
  Serial.println (buf);
} // end of printProgStr

void setup ()
{
  Serial.begin (115200);
  printProgStr (F("but does it get goat's blood out?"));
  Serial.println ();
}  // end of setup

void loop () {}

That buffers up the string so you can then do a single write on it (instead of the Serial.println that I have).

#17

Thanks, with a little bit of tweaking, that did it! I didn't realize there was a "pgm_read_byte()" function, that really helped.

#18