Pages: [1]   Go Down
Author Topic: Compress file before sending via WiFi/Ethernet possible?  (Read 582 times)
0 Members and 1 Guest are viewing this topic.
Central MN, USA
Offline Offline
Tesla Member
***
Karma: 64
Posts: 6889
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I have large files (several MB) and somehow my WiFi transfer is only up to 15KB/s. I wonder if doing DEFLATE compression is possible at all on Arduino:

http://en.wikipedia.org/wiki/DEFLATE

I guess not?!
Logged


0
Offline Offline
Tesla Member
***
Karma: 114
Posts: 8913
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If you are not currently doing so, you may want to build a large byte packet before sending similar to below. Currently you may be using an internet packet for each byte being sent.

Code:
//zoomkat 12/26/12
//SD server test code
//open serial monitor to see what the arduino receives
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields

#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = {
  192, 168, 1, 102 }; // ip in lan
byte gateway[] = {
  192, 168, 1, 1 }; // internet access via router
byte subnet[] = {
  255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
String readString;

//////////////////////

void setup(){

  Serial.begin(9600);

  // disable w5100 while setting up SD
  pinMode(10,OUTPUT);
  digitalWrite(10,HIGH);
  Serial.print("Starting SD..");
  if(!SD.begin(4)) Serial.println("failed");
  else Serial.println("ok");

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  //delay(2000);
  server.begin();
  Serial.println("Ready");
}

void loop(){
  // Create a client connection
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        //read char by char HTTP request
        if (readString.length() < 100) {
          //store characters to string
          readString += c;
          //Serial.print(c);
        }
        //if HTTP request has ended
        if (c == '\n') {

          ///////////////
          Serial.println(readString); //print to serial monitor for debuging

            client.println("HTTP/1.1 200 OK"); //send new page
          client.println("Content-Type: text/html");
          //client.println("Content-Type: image/jpeg");
          //client.println("Content-Type: image/gif");
          //client.println("Content-Type: application/x-javascript");
          //client.println("Content-Type: text");

          client.println();

          //File myFile = SD.open("boom.htm");
          File myFile = SD.open("HYPNO.JPG");
          //File myFile = SD.open("BLUEH_SL.GIF");
          //File myFile = SD.open("SERVOSLD.HTM");

          if (myFile) {

            byte clientBuf[64];
            int clientCount = 0;

            while(myFile.available())
            {
              clientBuf[clientCount] = myFile.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
            //final <64 byte cleanup packet
            if(clientCount > 0) client.write(clientBuf,clientCount);           
            // close the file:
            myFile.close();
          }
          delay(1);
          //stopping client
          client.stop();
          readString="";
        }
      }
    }
  }
}

Logged

Consider the daffodil. And while you're doing that, I'll be over here, looking through your stuff.   smiley-cool

Sweden
Offline Offline
Sr. Member
****
Karma: 6
Posts: 372
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I have large files (several MB) and somehow my WiFi transfer is only up to 15KB/s. I wonder if doing DEFLATE compression is possible at all on Arduino:
What type of data? There might be easier ways to compress it; Huffman, Run-length, DPCM, etc. Otherwise porting one of the many implementations of DEFLATE is an interesting challenge as they often use many times the memory (SRAM) available in a typical Arduino  ;-)

Cheers!
Logged

Central MN, USA
Offline Offline
Tesla Member
***
Karma: 64
Posts: 6889
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

The file is just text file with numbers (comma separated values):

Here is a sample line:

Code:
1363180287 "2013/3/13 13:11:27" 1223 32000 -1.51 -9.99 0 -0.02 0 0 0 0 0 0 0 5000 0 0 0 0 AR72000-0001 1

On another thought, since Arduino is able to act as a server, I can divide the data into one file per day and list all files so the person can download the right sets of data.
Logged


Sweden
Offline Offline
Sr. Member
****
Karma: 6
Posts: 372
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The file is just text file with numbers (comma separated values):

Here is a sample line:

Code:
1363180287 "2013/3/13 13:11:27" 1223 32000 -1.51 -9.99 0 -0.02 0 0 0 0 0 0 0 5000 0 0 0 0 AR72000-0001 1
Yes this is the type of text data that DEFLATE loves. Only a fraction of the ASCII symbols are used and there are many common sequences.

As the number of symbols are low you could use BCD for numbers and the extra 10..15 value for special characters and run-length of zero which is very common. This way the compression rate should be 3-2:1 with a very simple algorithm. This will require a 8-bit binary format and a decoding program on the "other size".

As the symbols are known a simple version of DEFLATE could be constructed that used a much smaller encoding table. But still the limited memory in Arduino is a problem. A Mega could give enough to work with (8 Kbyte).

Cheers!
Logged

Central MN, USA
Offline Offline
Tesla Member
***
Karma: 64
Posts: 6889
Phi_prompt, phi_interfaces, phi-2 shields, phi-panels
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks kowalski.

I dont know if I can send binary via http or only encoded binary, which robs away the compression gain.
Logged


Pages: [1]   Go Up
Jump to: