Ethernet, Free RAM and Buffered HTML requests

Over several months I have been building a sophisticated home automation system. I am using a Freetronics Ethermega card with 256K Flash, 8K RAM, 4K EEPROM. I am using its ethernet connectivity for the user interface and SD card file for non RAM data storage. The application is currently 5,000 lines and 97K.

On a few ocassions I have run out or RAM and the application has hung. Each time I have researched my options and applied fixes to reduce RAM usage. I use F() strings, have another 2.5K of strings in EEPROM and I am using the SD Card to store 3.6K of infrequently used strings that the application loads as required. The application (as of today) typically runs between 1K and 2K of free RAM depending on many factors.

I have written procedures to analyse and record the application's RAM utilisation during runtime and within selected (many) procedures. On occasions RAM has been seen to get very low, but apparently not run out because the application has continued to run. On some of these occasions my system's log files indicate large numbers of (generally php) hacking attempts directed to my external IP address which pass through to my application's ethernet server functionality via http port 80 . My system has recorded and successfully dealt with as many as 50 php hacking attempts within the space of one minute.

THE QUESTION:

Does anyone know the effect of buffered and unprocessed HTML (hack) requests on RAM within an Arduino ethernet unit? If many many HTML hack requests are sent to an IP address associated with an Arduino ethernet unit within a second will they quickly overflow RAM if the Arduino cannot dispatch (EthernetClient.flush() and EthernetClient.stop()) them faster than they are arriving? Or is it possible that my ADSL modem, ISP and the internet generally will buffer (or discard) excess requests until the Arduino ethernet unit is ready to process them.

I use the server code with a Mega/wethernet shield. I had buffer overflow problems that would crash my sketch. This is the code that now prevents that. I added a new variable to determine when the first line of the request ends.
http://playground.arduino.cc/Code/WebServerST

Here is the section of code. It stops the request buffer save at 63 characters. tBuf is a 64 character array.

    while (client.connected()) {
      while(client.available()) {
        // if packet, reset loopCount
        loopCount = 0;
        char c = client.read();

        if(currentLineIsGet && tCount < 63)
        {
          tBuf[tCount] = c;
          tCount++;
          tBuf[tCount] = 0;          
        }

I am not worried about processing a single html request header associated with an ethernet client connection. I use Strings (and sub stringing) to process the variable length lines (typically a few hundred bytes overall) within each header to extract the data I require (the URL page, cookies, host, passwords, etc).

What I am worried about are unprocessed html request headers which are accessed one-by-one via ethernet server.available(). If a hacking robot sends ten 300 byte html requests to me in less than one second where are they and is there a chance that 3,000 bytes of RAM will be consumed in an instant and cause my Arduino system to crash?

CatweazleNZ: I am not worried about processing a single html request header associated with an ethernet client connection. I use Strings (and sub stringing) to process the variable length lines (typically a few hundred bytes overall) within each header to extract the data I require (the URL page, cookies, host, passwords, etc).

What I am worried about are unprocessed html request headers which are accessed one-by-one via ethernet server.available(). If a hacking robot sends ten 300 byte html requests to me in less than one second where are they and is there a chance that 3,000 bytes of RAM will be consumed in an instant and cause my Arduino system to crash?

If you are using the String data type, you should be worried. I have never had good luck using them, and they have always managed to crash my sketches. Note I used a character array instead. I recommend you do the same.

You need not worry about multiple requests. They are stored in the w5100 socket buffers until you get around to them. There are only 4 sockets, so the most a bot could do is connect 4 times. Other attempts will not connect until you process one and free that socket for another connection.

Don’t know if it would address your concern, but the code I use seems to put a limit on the size of the String being captured.

if (readString.length() < 100) {
   readString += c;

SurferTim:
If you are using the String data type, you should be worried. I have never had good luck using them, and they have always managed to crash my sketches. Note I used a character array instead. I recommend you do the same.

You need not worry about multiple requests. They are stored in the w5100 socket buffers until you get around to them. There are only 4 sockets, so the most a bot could do is connect 4 times. Other attempts will not connect until you process one and free that socket for another connection.

I use Strings all the time, dozens, maybe 100 spread throughout my code with frequent use of String.indexOf() and String.substring() and I seem to have no problems. Over the course of a day I would do thousands of String operations without seeing any evidence of problem. Sometimes my system does run for days without problem but as I am still building it typically it is restarted multiple times a day for code changes that I upload.

I have extensive RAM checking within my application to monitor maximum stack size, heap size and the free heap list and well as tracking minimum RAM. This screenshot shows some of these stats after the application was running almost three hours. If the use of Strings does cause memory fragmentation I certainly do not see it. While my free RAM does vary between 3000 and 1000 bytes that is primarily related to the creation and destruction of ethernet client and SD card file objects I believe with only a small contribution from the transient use of Strings within called procedures.

CatweazleNZ

Strings.jpg

SurferTim: You need not worry about multiple requests. They are stored in the w5100 socket buffers until you get around to them. There are only 4 sockets, so the most a bot could do is connect 4 times. Other attempts will not connect until you process one and free that socket for another connection.

I think they can only queue up four html requests in the W5100 socket buffers. Using ethernet server.available() I process all html requests sequentially - so it looks like they can only consume one tranche of my system's RAM with the request content and other stuff associated with the ethernet client object while it exists.

Thanks for that. Now I understand why my system does not crash when I get the php bot attacks - sometimes 50 within a minute. (Well my application processes 50 in a minutes - excess ones may have been lost because they could not get into already full w5100 socket buffers.)

Catweazle NZ

IDE v1.0.5 is supposed to have fixed the String problem, so you may do ok. In a processor with this small amount of SRAM, I can allocate SRAM myself.

I would follow zoomkat's advice and limit the number of characters stored there.