Compress file before sending via WiFi/Ethernet possible?

I have large files (several MB) and somehow my WiFi transfer is only up to 15KB/s. I wonder if doing DEFLATE compression is possible at all on Arduino:

I guess not?!

If you are not currently doing so, you may want to build a large byte packet before sending similar to below. Currently you may be using an internet packet for each byte being sent.

//zoomkat 12/26/12
//SD server test code
//open serial monitor to see what the arduino receives
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields

#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 
  192, 168, 1, 102 }; // ip in lan
byte gateway[] = { 
  192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 
  255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
String readString; 

//////////////////////

void setup(){

  Serial.begin(9600);

  // disable w5100 while setting up SD
  pinMode(10,OUTPUT);
  digitalWrite(10,HIGH);
  Serial.print("Starting SD..");
  if(!SD.begin(4)) Serial.println("failed");
  else Serial.println("ok");

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  //delay(2000);
  server.begin();
  Serial.println("Ready");
}

void loop(){
  // Create a client connection
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        //read char by char HTTP request
        if (readString.length() < 100) {
          //store characters to string 
          readString += c; 
          //Serial.print(c);
        } 
        //if HTTP request has ended
        if (c == '\n') {

          ///////////////
          Serial.println(readString); //print to serial monitor for debuging 

            client.println("HTTP/1.1 200 OK"); //send new page
          client.println("Content-Type: text/html");
          //client.println("Content-Type: image/jpeg");
          //client.println("Content-Type: image/gif");
          //client.println("Content-Type: application/x-javascript");
          //client.println("Content-Type: text");

          client.println();

          //File myFile = SD.open("boom.htm");
          File myFile = SD.open("HYPNO.JPG");
          //File myFile = SD.open("BLUEH_SL.GIF");
          //File myFile = SD.open("SERVOSLD.HTM");

          if (myFile) {

            byte clientBuf[64];
            int clientCount = 0;

            while(myFile.available())
            {
              clientBuf[clientCount] = myFile.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
            //final <64 byte cleanup packet
            if(clientCount > 0) client.write(clientBuf,clientCount);            
            // close the file:
            myFile.close();
          }
          delay(1);
          //stopping client
          client.stop();
          readString="";
        }
      }
    }
  } 
}

liudr:
I have large files (several MB) and somehow my WiFi transfer is only up to 15KB/s. I wonder if doing DEFLATE compression is possible at all on Arduino:

What type of data? There might be easier ways to compress it; Huffman, Run-length, DPCM, etc. Otherwise porting one of the many implementations of DEFLATE is an interesting challenge as they often use many times the memory (SRAM) available in a typical Arduino :wink:

Cheers!

The file is just text file with numbers (comma separated values):

Here is a sample line:

1363180287	 "2013/3/13 13:11:27"	1223	32000	-1.51	-9.99	0	-0.02	0	0	0	0	0	0	0	5000	0	0	0	0	 AR72000-0001	1

On another thought, since Arduino is able to act as a server, I can divide the data into one file per day and list all files so the person can download the right sets of data.

liudr:
The file is just text file with numbers (comma separated values):

Here is a sample line:

1363180287	 "2013/3/13 13:11:27"	1223	32000	-1.51	-9.99	0	-0.02	0	0	0	0	0	0	0	5000	0	0	0	0	 AR72000-0001	1

Yes this is the type of text data that DEFLATE loves. Only a fraction of the ASCII symbols are used and there are many common sequences.

As the number of symbols are low you could use BCD for numbers and the extra 10..15 value for special characters and run-length of zero which is very common. This way the compression rate should be 3-2:1 with a very simple algorithm. This will require a 8-bit binary format and a decoding program on the "other size".

As the symbols are known a simple version of DEFLATE could be constructed that used a much smaller encoding table. But still the limited memory in Arduino is a problem. A Mega could give enough to work with (8 Kbyte).

Cheers!

Thanks kowalski.

I dont know if I can send binary via http or only encoded binary, which robs away the compression gain.