EOF --Stop HTML code from being displayed in text file

I have a webserver that servers a web page with a link to open or save a file. Link appears to be working correctly; with one exception, before the download returns to the web page, it appends HTML document to the donloaded text file.

03/26/2014 , 13:45:00,37.70 %, 66.56  F., 29.33 in. Hg. , 993.09 millibars , 0.98, 565.01 Ft. 
03/26/2014 , 14:00:00,37.40 %, 66.38  F., 29.32 in. Hg. , 992.88 millibars , 0.98, 571.04 Ft. 
03/26/2014 , 14:15:00,37.50 %, 66.56  F., 29.31 in. Hg. , 992.51 millibars , 0.98, 581.76 Ft. 
03/26/2014 , 14:30:00,37.60 %, 66.74  F., 29.31 in. Hg. , 992.44 millibars , 0.98, 583.74 Ft. 
03/26/2014 , 14:45:00,37.30 %, 66.74  F., 29.30 in. Hg. , 992.20 millibars , 0.98, 590.75 Ft. 
03/26/2014 , 15:00:00,36.90 %, 66.92  F., 29.29 in. Hg. , 991.83 millibars , 0.98, 601.28 Ft. 
03/26/2014 , 15:15:00,36.50 %, 66.92  F., 29.28 in. Hg. , 991.63 millibars , 0.98, 607.16 Ft. 
03/26/2014 , 15:30:00,36.70 %, 67.28  F., 29.28 in. Hg. , 991.50 millibars , 0.98, 611.03 Ft. 
03/26/2014 , 15:45:00,36.80 %, 67.28  F., 29.27 in. Hg. , 991.16 millibars , 0.98, 620.94 Ft. 
03/26/2014 , 16:00:00,36.90 %, 67.46  F., 29.26 in. Hg. , 990.78 millibars , 0.98, 632.08 Ft. 
????????????????????????????????????????????????????????HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 467
Connection: close
Refresh: 30

<!DOCTYPE HTML>
<html>
<h2>Weather Observations</h2>
Treyburn Lakes
Indianapolis, IN 46239
Date,Time:  03/26/2014 , 19:28:25 EST
Temperature:  67.46 Deg. F.
Humidity:  37.00 %
Barometric Pressure:  29.23 in. Hg.
Barometric Pressure:  989.98 mb.
Atmosphere:  0.98 Atm
Altitude:  654.46 Feet


<h2>Collected Observations</h2><a href='log.txt' download='log.txt' >Download: log.txt</a>


</html>

Here is code for the download:

if (StrContains(HTTP_req, "GET /log.txt")) {
            
				server.println("HTTP/1.1 200 OK");
				server.println("Content-Type: text/plain; charset = UTF-8");
				server.println("Content-Disposition: attachment");
				server.println("Content-Length: ");
				server.println();
				
			webFile = SD.open("log.txt");
			
			if(webFile){
			
			  while(webFile.available()){ //read till file end is reached
			     for (int i = 0;i < 93;i++){ //read 93 bytes into buffer before sending
				 buffer[i] = webFile.read();
                 }
				 server.write(buffer,93);  //send file to browser
				 } //end while (webFile.available())
				 webFile.close();
				 currentLineIsBlank = false;
				 } //end if (webFile)
				 else{
				 Serial.println("File error");
				 }
          }

Is there a way to check for the occurrence of “FF.” Unused space is marked FF looking at the webFile with a Hex editor.
Or is there another way to detect End of File?

while(webFile.available()){ //read till file end is reached
			     for (int i = 0;i < 93;i++){ //read 93 bytes into buffer before sending
				 buffer[i] = webFile.read();
                 }

While there
is at least one character
left in the file,
read all 93 characters.

When the hell are you planning to learn to properly indent your code?

@PaulS

Do you have a solution?

My Disability sometimes interrupts my thoughts; I do my best… sometimes I need help.

Wm.

webFile records are 93 bytes long, after the last record all remaining bytes in the file are FF.

I am trying to find how to detect the presence of "FF" to prevent the buffer from continuing to over flow --at least
this is what I think is happening..

Wm.

Maybe the issue is the buffer is not null-terminated. I am sure there is an obvious solution, but I have not worked in this area of Arduino...

Maybe one of our experienced HTML/arduino members will have the exact solution... Friday night during basketball playoffs is a bad time to have code issues :smiley:

Ray

Does the file actually end at 93 bytes? Your code assumes that there is always an exact multiple of 93 characters in the file, but I have no idea whether this is a valid assumption. It’s not a desirable one in any case. If you’re reading the whole file, why don’t you just keep reading until there are no more bytes available - I don’t see any need for the double loops.

I haven’t followed how the snippet you posted would lead to a second HTTP header being appended to the response, but it’s hard to understand these snippets in isolation - much easier if you posted the whole sketch.

Some code that may have what you need. The file read/upload has a clean up section for the last bytes in the file.

//zoomkat 12/26/12
//SD server test code
//open serial monitor to see what the arduino receives
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields

#include <SD.h>
#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 
  192, 168, 1, 102 }; // ip in lan
byte gateway[] = { 
  192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 
  255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
String readString; 

//////////////////////

void setup(){

  Serial.begin(9600);

  // disable w5100 while setting up SD
  pinMode(10,OUTPUT);
  digitalWrite(10,HIGH);
  Serial.print("Starting SD..");
  if(!SD.begin(4)) Serial.println("failed");
  else Serial.println("ok");

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  //delay(2000);
  server.begin();
  Serial.println("Ready");
}

void loop(){
  // Create a client connection
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        //read char by char HTTP request
        if (readString.length() < 100) {
          //store characters to string 
          readString += c; 
          //Serial.print(c);
        } 
        //if HTTP request has ended
        if (c == '\n') {

          ///////////////
          Serial.println(readString); //print to serial monitor for debuging 

            client.println("HTTP/1.1 200 OK"); //send new page
          //client.println("Content-Type: text/html");
          client.println("Content-Type: image/jpeg");
          //client.println("Content-Type: image/gif");
          //client.println("Content-Type: application/x-javascript");
          //client.println("Content-Type: text");

          client.println();

          //File myFile = SD.open("boom.htm");
          File myFile = SD.open("HYPNO.JPG");
          //File myFile = SD.open("BLUEH_SL.GIF");
          //File myFile = SD.open("SERVOSLD.HTM");

          if (myFile) {

            byte clientBuf[64];
            int clientCount = 0;

            while(myFile.available())
            {
              clientBuf[clientCount] = myFile.read();
              clientCount++;

              if(clientCount > 63)
              {
                // Serial.println("Packet");
                client.write(clientBuf,64);
                clientCount = 0;
              }
            }
            //final <64 byte cleanup packet
            if(clientCount > 0) client.write(clientBuf,clientCount);            
            // close the file:
            myFile.close();
          }
          delay(1);
          //stopping client
          client.stop();
          readString="";
        }
      }
    }
  } 
}

@PeterH

Buffered_www.ino is attached. Each record length of webFile is 93 characters. webFile record gets updated every 15 minutes.
Thank you PeterH.

@zoomkat

Will try your code -think I have tried it already after searching arduino.cc forum. Will post result of trying your code.
Thank you zoomkat.

Buffered_www.ino (15.6 KB)

@zoomkat

Copied and pasted file handling code in to Buffered_www.ino sketch. Same result -webFile downloads with HTML document appended to the end of webFile.

Wm.

Copied and pasted file handling code in to Buffered_www.ino sketch.

So, you changed your code, and you want us to guess why your modified code doesn't work. I'll pass.

I am not using the Arduino Ethernet library --wireless networking is built in the Digistump Digix development board and uses the DigiFi library; therefore, I had no choice but to test this way. Thank you PaulS.

Digistump Digix is an Arduino compatible development board digix:overview [Digistump Wiki]

I am still trying to use the code I uploaded earlier Buffered_www.ino.
Wm.

If you can find what the below byte is, then maybe use is a delimiter to indicate the end of the file. Using the below code I sent the ? to the arduino using the serial monitor (copy and ctrl-v in text box) and got back 63 and 3f.

?
byte INFO;

void setup (){
  Serial.begin (9600);
}

void loop ()
{
  if(Serial.available ())  
  {
    INFO = Serial.read();
    Serial.println(INFO, DEC);
    Serial.println(INFO, HEX);
    Serial.println();
  }
}

@zoomkat

I confirm your results with the Digix Development board: 63 and 3F

Wm

Techno500:
webFile records are 93 bytes long, after the last record all remaining bytes in the file are FF.

I am trying to find how to detect the presence of "FF" to prevent the buffer from continuing to over flow --at least
this is what I think is happening..

Wm.

You tried to read 93 chars when you determined that at least 1 was available.
But even a 16 MHz AVR is faster than the data comes in while your ARM board is 3x faster than that.
When you read FF, that is 8 bit signed for -1. -1 means no char was available. Legal chars are 0 to 127.

If you're going to buffer then process the thing to do is have one task that collects chars as they become available, checks them for validity and buffers the ones you want then sets a flag when you have your whole line.
That task should be at the top of loop and loop should run around and round just checking for available characters until the buffer is ready to process. Don't make a loop inside of loop() to get the 93, just keep count as you read and buffer and use the count and the chars to know when to set the flag or handle an error. Yes, error. Handle possible errors or the first one you get (transmitted data is not guaranteed) will likely be a real pain.

Under the if available { } block, put the process the buffer inside of an if buffer ready { } block. When the flag is set, this code runs and saves the line or whatever you're doing with the data. When it's done, clear the flag.

Then loop() runs around and round until the buffer is ready again, the flag is set and the process code gets run.

Another thing to keep in mind is that readBytes() will read up to the specified number of bytes AND tell you how many it was actually capable of reading, so you don't ASSUME that there were 93 characters read, when in reality it might have been 3.

03/29/2014 , 16:45:00,40.50 %, 66.20  F , 29.11 in. Hg. , 985.77 millibars , 0.97, 773.19 Ft. 
03/29/2014 , 18:45:00,40.00 %, 66.02  F., 29.17 in. Hg. , 987.74 millibars , 0.97, 717.43 Ft. 
03/29/2014 , 19:00:00,41.50 %, 66 74  F., 29.17 in. Hg. , 987.80 millibars , 0.97, 715.91 Ft. 
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿ = alt 152 Extended Ascii character

From Google search for ÿ code point 0xFF:

“ÿ is the glyph for the character which (in Unicode and many ISO-8859-? encodings) has the ordinal value 0xFF. That value, also known in decimal as 255, is also used in some contexts as “the end-of-file character” (aka EOF) – although there’s no standard that defines the character as such (AFAIK), the value -1 is what does get returned in many languages (such as C) when you try to read more from a file that’s exhausted (“at end of file”).”

“In practice, therefore, an unexpected ÿ in your output often means that you are erroneously interpreting a byte that’s intended to signify “end of something” (a byte encoded with all bits set to one) as if it was part of the text to display.”

I have elimated the HTML code that was appended to webFile by declaring charset encoding for HTML dynamic web page and the download. All that is left now is the line of ÿ Extended Ascii characters that is somehow appended to the end of the webFile.

testing_buffered_file_read.ino (15.7 KB)

So don’t buffer them.
Check what you read as you read it and don’t buffer “illegal” chars.

If you want to be really sure what chars are what, make a sketch to print an ASCII table since the extended characters are not always displayed the same.

This is the Arduino ASCII Table sketch, it prints char 33 to char 126 which are the printable text chars.
32 is space, 10 is newline, 13 is linefeed

For your code, legal is probably 32 to 126 inclusive, but you need to check that.

/*
  ASCII table
 
 Prints out byte values in all possible formats:  
 * as raw binary values
 * as ASCII-encoded decimal, hex, octal, and binary values
 
 For more on ASCII, see http://www.asciitable.com and http://en.wikipedia.org/wiki/ASCII
 
 The circuit:  No external hardware needed.
 
 created 2006
 by Nicholas Zambetti 
 modified 9 Apr 2012
 by Tom Igoe
 
 This example code is in the public domain.

 <http://www.zambetti.com> 
 
 */
void setup() { 
 //Initialize serial and wait for port to open:
  Serial.begin(9600); 
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }
  
  // prints title with ending line break 
  Serial.println("ASCII Table ~ Character Map"); 
} 

// first visible ASCIIcharacter '!' is number 33:
int thisByte = 33; 
// you can also write ASCII characters in single quotes.
// for example. '!' is the same as 33, so you could also use this:
//int thisByte = '!';  

void loop() { 
  // prints value unaltered, i.e. the raw binary version of the 
  // byte. The serial monitor interprets all bytes as 
  // ASCII, so 33, the first number,  will show up as '!' 
  Serial.write(thisByte);    

  Serial.print(", dec: "); 
  // prints value as string as an ASCII-encoded decimal (base 10).
  // Decimal is the  default format for Serial.print() and Serial.println(),
  // so no modifier is needed:
  Serial.print(thisByte);      
  // But you can declare the modifier for decimal if you want to.
  //this also works if you uncomment it:

  // Serial.print(thisByte, DEC);  


  Serial.print(", hex: "); 
  // prints value as string in hexadecimal (base 16):
  Serial.print(thisByte, HEX);     

  Serial.print(", oct: "); 
  // prints value as string in octal (base 8);
  Serial.print(thisByte, OCT);     

  Serial.print(", bin: "); 
  // prints value as string in binary (base 2) 
  // also prints ending line break:
  Serial.println(thisByte, BIN);   

  // if printed last visible character '~' or 126, stop: 
  if(thisByte == 126) {     // you could also use if (thisByte == '~') {
    // This loop loops forever and does nothing
    while(true) { 
      continue; 
    } 
  } 
  // go on to the next character
  thisByte++;  
}

You need to be a bit careful when you are talking about an "FF" character, because that could mean different things.

There is 0xFF, which is binary 11111111 , equivalent to the integer value -1

and there is 0x0C, which is binary 00001100, which is "FF", the form-feed character, which is used to advance to the next page in old-style printers, and has been used with other meanings in various obsolete text file formats.

I read about halfway through this thread before I got confused about which FF you were talking about.

Techno500:
Is there a way to check for the occurrence of "FF." Unused space is marked FF looking at the webFile with a Hex editor.