Internet download of Arduino SD Card file

Trying to come up with a way to prevent a failed download from “hanging” sketch. So far, I have only thought about running a timer in a loop to allow download until time elapses; then to exit and close the open file and close connection.

Any suggestions?

else if (StrContains(HTTP_req, "GET /log.txt"))
                    {

                        if (    (((RTCTimedEvent.time.minute) == 0)||
                                 ((RTCTimedEvent.time.minute) == 15)||
                                 ((RTCTimedEvent.time.minute) == 30)||
                                 ((RTCTimedEvent.time.minute) == 45))
                                && ((RTCTimedEvent.time.second) == 00))
                        {
                            monitor;  //give priority to data logging
                        }

                        webFile = SD.open("log.txt");

                        delay(2000);

                        long int web =  webFile.size();
                        Serial.print("File size: ");
                        Serial.print(web);
                        Serial.println("");

                        if (webFile)
                        {

                            client.println("HTTP/1.1 200 OK");
                            client.println("Content-Type: application/octet-stream");
                            client.println("Content-Disposition: attachment");
                            client.println("Content-Length: , web");
                            client.println("Connnection: close");
                            client.println();

                            //this goes in the function that sends the data
                            char buffer[ BUFSIZE ];
                            int i;

                            while ( webFile.available() )
                            {
                                for (i=0 ; (i < BUFSIZE) && (buffer[ i-1 ] >= 0) ; i++)
                                {
                                    buffer[i] = webFile.read();
                                }
                                client.write( buffer, (buffer[i-1] >= 0) ? i : i-1 );
                            }
                        }

                        Serial.println("Ready");
                        Serial.end();

                        webFile.close();
                        delay(30);


                        // reset buffer index and all buffer elements to 0
                        req_index = 0;
                        StrClear(HTTP_req, REQ_BUF_SZ);

                    }

William

Download section.zip (769 Bytes)

What are you downloading from and to?

You can set up a timeout but an Arduino cannot reset a serial port on a PC - that has to happen on the PC.

…R

The file lives on SD card on the CC3000 shield. File is a collection of data from a dynamic web page that is generated by sketch running on the Arduino Mega and outputted as text with a download link to the Data collection.

Download link works --if the download starts; for whatever reason, if the download fails to completely download, the sketch “hangs.”

Attaching complete sketch, note requires Arduino Mega due to memory requirements.

William

CC3000_Weather_Observations_Webserver.zip (9.07 KB)

If you mean you are having a problem with downloading from the web I think you should put that in your title. You can change the title by modifying your original post.

I would have to learn about web stuff myself, so I can't offer immediate advice.

...R

Sometimes the download fails to start, HTTP request is good, clicking the link in the client browser produces a pop-up "Open as/Save as" window. All I see when I look at Download in Firefox 35.0.1 is "Unknown time and 0 bytes.

Execution of the sketch stops at this point and Serial Monitor has to be restarted; before new connections to web server can be completed.

Is there a way to test for failed download?

How can I detect a failed read of the data off the SD Card?

See previous reply for complete Arduino Mega sketch.

Best Regards, William

Techno500: Any suggestions?

You are not really interested in seeing the "Ready" message in the Serial monitor, aren't you?

 Serial.println("Ready");  // Push "Ready" into the Serial send buffer (not sent yet)
 Serial.end(); // Shut down Serial and throw away all contents from the send buffer

Serial is interrupt driven and these two lines of Code will do that, what I have written as comments.

If you want to see the "Ready" message on your Serial monitor, you must delay the "Serial.end();" command, until everything is sent. This can be done by a "Serial.flush()" command.

So if you like to see the "Ready" message, don't shut down Serial before everything is actually sent!

 Serial.println("Ready");  // Push "Ready" into the Serial send buffer (not sent yet)
 Serial.flush(); // Wait until everything in the send buffer has been sent
 Serial.end(); // Shut down Serial

Debugging code with Serial messages can become a nightmare if you send a message and then shut down Serial before the message is actually sent.

Why do you shut down Serial at all in your program?

Okay on the “Serial.flush()” issue.

Back to topic:

if(webFile)
						{
						
			    client.println("HTTP/1.1 200 OK");
                            client.println("Content-Type: application/octet-stream");
                            client.println("Content-Disposition: attachment");
                            client.println("Content-Length: , web");
                            client.println("Connnection: close");
                            client.println();
							
			    char buffer[ BUFSIZE ];
                            int i;

                            while ( webFile.available() )
                            {
                                for (i=0 ; (i < BUFSIZE) && (buffer[ i-1 ] >= 0) ; i++)
                                {
                                    buffer[i] = webFile.read();
																		
                                }
                                client.write( buffer, (buffer[i-1] >= 0) ? i : i-1 );
								
                            }
			exit;	
                        }



						
                        // reset buffer index and all buffer elements to 0
                        req_index = 0;
                        StrClear(HTTP_req, REQ_BUF_SZ);
						
			// give the web browser time to receive the data
			webFile.close();
                       	// close the connection:
			client.close();
			delay(1000);
			Serial.println("Client connection closed");
			Serial.println("");
			Serial.flush();
                        Serial.end();
                        exit;
                        }

How can I check the “buffer_” to see if it is empty (0 bytes)? If “buffer*” is empty I would like to exit before the client.write.*_

Code that can send a fairly large jpg picture to an browser.

//zoomkat 12/26/12
//SD server test code
//open serial monitor to see what the arduino receives
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields

#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 
  192, 168, 1, 102 }; // ip in lan
byte gateway[] = { 
  192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 
  255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
String readString; 

//////////////////////

void setup(){

  Serial.begin(9600);

  // disable w5100 while setting up SD
  pinMode(10,OUTPUT);
  digitalWrite(10,HIGH);
  Serial.print("Starting SD..");
  if(!SD.begin(4)) Serial.println("failed");
  else Serial.println("ok");

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  //delay(2000);
  server.begin();
  Serial.println("Ready");
}

void loop(){
  // Create a client connection
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        //read char by char HTTP request
        if (readString.length() < 100) {
          //store characters to string 
          readString += c; 
          //Serial.print(c);
        } 
        //if HTTP request has ended
        if (c == '\n') {

          ///////////////
          Serial.println(readString); //print to serial monitor for debuging 

            client.println("HTTP/1.1 200 OK"); //send new page
          //client.println("Content-Type: text/html");
          client.println("Content-Type: image/jpeg");
          //client.println("Content-Type: image/gif");
          //client.println("Content-Type: application/x-javascript");
          //client.println("Content-Type: text");

          client.println();

          //File myFile = SD.open("boom.htm");
          File myFile = SD.open("HYPNO.JPG");
          //File myFile = SD.open("BLUEH_SL.GIF");
          //File myFile = SD.open("SERVOSLD.HTM");

          if (myFile) {

            byte clientBuf[64];
            int clientCount = 0;

            while(myFile.available())
            {
              clientBuf[clientCount] = myFile.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
            //final <64 byte cleanup packet
            if(clientCount > 0) client.write(clientBuf,clientCount);            
            // close the file:
            myFile.close();
          }
          delay(1);
          //stopping client
          client.stop();
          readString="";
        }
      }
    }
  } 
}

@zoomkat Thank you for your code example; for clarification, download code I am using is working and has been optimized for the "CC3000 Wifi Shield" a product of Adafruit. Only on an occasion, a download will fail with 0 bytes.

I am looking for a way to gracefully exit the prior to the actual download if the read buffer has 0 bytes. This causes the sketch to hang with a reset of serial monitor. This re-initializes the "CC3000 Wifi Shield." Download capability is restored.

I would like to eliminate the need to reset Serial Monitor; for better reliability.

William

This causes the sketch to hang with a reset of serial monitor.

The code that you didn't post has problems. Hmmm, what to do?

Techno500:
I am looking for a way to gracefully exit the prior to the actual download if the read buffer has 0 bytes. This causes the sketch to hang with a reset of serial monitor. This re-initializes the “CC3000 Wifi Shield.”

When your sketch magically restarts while at work, this is a relatively clearly sign for either

  • invalid pointer operations writing to the wrong positions in RAM
  • or your sketch tries using more RAM than the controller has built-in
    (which leads to the same problems)

Did you a check of “free RAM” anywhere in your sketch?
Result of “free RAM” check?

If RAM would be a problem, I’d at first change all print/println commands with string constants as the parameter to using the F-macro:

client.println("HTTP/1.1 200 OK");
client.println("Content-Type: application/octet-stream");
client.println("Content-Disposition: attachment");
client.println("Content-Length: , web");
client.println("Connnection: close");

becomes using the F-macro saving many bytes of RAM:

client.println(F("HTTP/1.1 200 OK"));
client.println(F("Content-Type: application/octet-stream"));
client.println(F("Content-Disposition: attachment"));
client.println(F("Content-Length: , web"));
client.println(F("Connnection: close"));

Next thing then would be the file reading logic while sending the file to the client.

P.S.: What’s this you send:
client.println(F(“Content-Length: , web”));
You send the string “, web” as the number of bytes you will send?
I think after sending "Content-Length: " there should follow a number and not “, web”.

@PaulS you missed the complete code in the first message and the code exert in a previous message in this topic thread; presented here again…

@jurs Thank you; I will make note of your coding suggestion. Memory should not be an issue; I forgot to mention, I am using an “Arduino Mega 2560.”

I will try using the “webFile.size()” function with Content-Length instead of using web. Thank you.

Code that was in previous message:

if(webFile)
	    {
	    		    client.println("HTTP/1.1 200 OK");
                            client.println("Content-Type: application/octet-stream");
                            client.println("Content-Disposition: attachment");
                            client.println("Content-Length: , web");
                            client.println("Connnection: close");
                            client.println();
							
			    char buffer[ BUFSIZE ];
                            int i;

                            while ( webFile.available() )
                            {
                                for (i=0 ; (i < BUFSIZE) && (buffer[ i-1 ] >= 0) ; i++)
                                {
                                    buffer[i] = webFile.read();
																		
                                }
                                client.write( buffer, (buffer[i-1] >= 0) ? i : i-1 );
								
                            }
			 exit;	
                        }



						
                        // reset buffer index and all buffer elements to 0
                        req_index = 0;
                        StrClear(HTTP_req, REQ_BUF_SZ);
						
			// give the web browser time to receive the data
			webFile.close();
                       	// close the connection:
			client.close();
			delay(1000);
			Serial.println("Client connection closed");
			Serial.println("");
			Serial.flush();
                        Serial.end();
                        exit;
                        }

Best Regards,
William

Techno500:
Memory should not be an issue; I forgot to mention, I am using an “Arduino Mega 2560.”

OK, if RAM is no problem, then the next thing would be the programming logic how you read the file into a buffer. This will do 64 single byte reads:

for (i=0 ; (i < BUFSIZE) && (buffer[ i-1 ] >= 0) ; i++)
{
  buffer[i] = webFile.read();
}

As the SD card and the Ethernet shield are on the same SPI bus, this will (possible, I’m not quite sure) mean:

  • 64 times switch from Ethernet-SPI to SD-SPI
  • 64 times read a single byte from SD into buffer
  • 64 times switch from SD-SPI to Ethernet-SPI

And if it not so, it will at least make it slow sending the file.

So if your buffer is 64 bytes in size, you’d better read the whole buffer at once from SD card (which you are doing NOT), as well as you are sending the whole buffer at once to the Ethernet shield (which you are actually doing). So your programming logic “reading buffer from SD → sending buffer to client” needs a rewrite.

Techno500:
I will try using the “webFile.size()” function with Content-Length instead of using web.

I think this will be much better conform to the Internet standards, sending the file size as “Content-Length” and not some bogus string.

BTW: Does your program possibly use ‘String’ objects in any place of your program?
Or do you just work with char arrays?

I am looking for a way to gracefully exit the prior to the actual download if the read buffer has 0 bytes. This causes the sketch to hang with a reset of serial monitor.

You might look at the peek function to determine if the incoming buffer has any bytes, and if not, close the connection. That being said, I would think that a byte/bytes arriving in the input buffer is the indication that a client is trying to connect.

http://arduino.cc/en/Serial/Peek

                            client.println("Content-Length: , web");

Nonsense. The content length statement actually tells the browser how big the file is. ", web" is NOT a size!

Thank you jurs, zoomkat, and paulS.

@jurs Buffer on the CC3000 works best and the fastest thoughput at 32 Bytes. I will test with 64 bytes.

@zoomkat Will give it a try.

@paulS Thank you, I missed that one. Good to have a fresh set of eyes!

William