Off-by one with SD.read()

In the following code, the 'else' statement inside a while(1) loop ran when it wasn't suppose to.

The following code reads a text file called "Test_1v2.txt" which contains 36 characters + 1 newline character = 37 total characters. The while loop should iterate 37 times.

On the 37th iteration, file.read() should assign newChar with an endline character. The if(newChar == '\n') should be true and we should have exit from the while loop. Instead the if() condition FAILS and execution falls through to the else statement. One iteration later, we see that execution THEN breaks out of the code.

#include <SPI.h>
#include <SD.h>

void p1(char string[], int val){Serial.print(string);Serial.print(":");Serial.println(val);} 
void p2(char string[], char a){Serial.print(string);Serial.write(a);Serial.println();}

File file; 

void setup() {
 Serial.begin(9600); 
 if (!SD.begin(4)){Serial.println("initialization failed!"); return;} 
 
 Serial.println("Program start."); 
 
 /*file = SD.open("Test_1v3.txt", FILE_READ); 
 if(file){
   char newChar; 
   int numNewline = 0; 
   while(1){
     newChar = file.read();
     if(newChar == '\n'){
        numNewline++;  
     }
     if(newChar == EOF)
       break; 
     else 
       Serial.print(newChar); 
   }
   p1("numNewline", numNewline); 
   
 }else Serial.println("Could not open file for reading."); 
 file.close();*/
 
 /*
 //Moving around using seek() and read() to println
 file = SD.open("Test_1v3.txt", FILE_READ);
 if(file){
   file.seek(36); //Put cursor BEFORE first endline. 789|\n
   Serial.write(file.read()); //Print endline
   
   file.seek(0); 
   Serial.write(file.read()); 
   
   //file.seek(37); 
   //Serial.write(file.read());
 }else Serial.println("Could not open file for writing."); 
 file.close(); */
 
 //Count the number of characters.
 file = SD.open("Test_1v2.txt", FILE_READ); 
 if(file){
   char newChar; 
   int counter = 0;     
   int numEndlines = 0; 
   
   file.seek(0); 
   while(1){
     newChar = file.read();
     if(newChar == '\n'){
        numEndlines++; 
        Serial.println("Newline reached! Breaking out of while loop before printing \\n!");
        p1("numEndlines", numEndlines);
        break;
     }
     else{
       p2("newChar = ", newChar);
       counter++; 
       p1("Counter", counter); 
     } 
     if(newChar == 'EOF')
       Serial.println("EOF Reached"); 
   }
   
 }else Serial.println("Could not open file for reading."); 
 file.close(); 

 Serial.println("Done."); 
}

void loop() {
}

Test_1v2.txt (40 Bytes)

EX_SDSeek.ino (1.93 KB)

Please post your code here (using code tags) to avoid the need to download it in order to provide help.

...I expect the while loop to run 37 time...

There is no "the while loop". Your code contains two of them.

You must read the first character before the while-loop, otherwise you are testing an
uninitialized variable.

The first and second File IO's are irrelevant. The third, with while(1) is what this question references.
They do not affect how the third while loop runs.

I updated my op to include what is only relevant.

MarkT:
You must read the first character before the while-loop, otherwise you are testing an
uninitialized variable.

I'm certain this isn't the case:

    char newChar;     //Declared but Uninitialized
    int counter = 0; 
    file.seek(0); 

    while(1){           //Run infinitely
      newChar = file.read();  //Initialized newChar
      if(newChar == '\n')      //Test the newly initialized
         break;
      else{
        counter++; 
      } 
    }

Also, if was was as you've said, then I would have printed out garbage in the first iteration and not the letter 'a' (which is what gets printed).

The odd behavior seems to be that the if(newChar == '\n') condition fails when I expect newChar to contain newline. So either newChar doesn't contain a newline, or the textfile doesn't contain a new line, or there is a subtlety to read() that I'm missing.

I'll test to see that textfile does have a new line and that newChar has newline.

But to verify read() works like this right? (Let '|' denote where the read cursor is and calling read() makes the cursor move 1 character to the right and returning the character it passes.)

|0123456789\n //Original
0123456789|\n //After calling seek(10)
0123456789\n| //After calling read(), '\n' gets returned

The odd behavior seems to be that the if(newChar == '\n') condition fails when I expect newChar to contain newline.

What evidence have you got for this ? What does the output look like ?

or the textfile doesn't contain a new line,

It certainly isn't as you describe in your original post.

abcdefghijklmnopqrstuvwxyz0123456789\n\n

After the alphabet and the numbers there are two Carriage Return/Linefeed pairs in the file. So, before it gets to the Linefeed it prints the Carriage Return using your P2 function and that inserts the linefeed that you are presumably seeing because of the Serial.println() in the function.

UKHeliBob:
It certainly isn't as you describe in your original post.

I've edited my OP to reflect changes.

UKHeliBob:
After the alphabet and the numbers there are two Carriage Return/Linefeed pairs in the file. So, before it gets to the Linefeed it prints the Carriage Return using your P2 function and that inserts the linefeed that you are presumably seeing because of the Serial.println() in the function.

When I removed Serial.println() in P2 function, the output (output2) shows that a newline was not printed (which is the behavior we want). But on the 37th iteration, I did not expect to see the 3 statements in the body of the else{} statement run and it did run!

Tweaking a little bit more: I'm not sure what gets printed on the 37th iteration ...

Note: Despite what the output says, 38 iteration takes place. On the 38th iteration, the newline character is recognized, execution goes to the if() statement and we break out of the loop before numIteration gets incremented. So on the 38th iteration, we break.

Breaking was suppose to occur on the 37th iteration with the iteration count being 36.

#include <SPI.h>
#include <SD.h>

void p1(char string[], int val){Serial.print(string);Serial.print(":");Serial.println(val);} 
void p2(char string[], char a){Serial.print(string);Serial.write(a); Serial.print(" ");}

File file; 

void setup() {
  Serial.begin(9600); 
  if (!SD.begin(4)){Serial.println("initialization failed!"); return;} 
  
  Serial.println("Program start."); 
  
  //Count the number of characters.
  file = SD.open("Test_1v2.txt", FILE_READ); 
  if(file){
    char newChar; 
    int counter = 0;     
    int numEndlines = 0; 
    int numIterations = 0; 
    
    file.seek(0); 
    while(1){
      Serial.println("Getting newChar"); 
      newChar = file.read();
      Serial.println(newChar); 
      
      if(newChar == '\n'){
         Serial.println("Running if"); 
         numEndlines++; 
         Serial.println("Newline reached! Breaking out of while loop before printing \\n!");
         p1("numEndlines", numEndlines);
         break;
      }
      else{
        Serial.println("Running else"); 
        p2("newChar = ", newChar);
        counter++; 
        p1("Counter", counter); 
      } 
      if(newChar == 'EOF')
        Serial.println("EOF Reached"); 
        
      numIterations++; 
      p1("numIterations",numIterations); 
      Serial.println(); 
    }
      p1("TOTAL numIterations",numIterations); 
  }else Serial.println("Could not open file for reading."); 
  file.close(); 
 
  Serial.println("Done."); 
}

void loop() {
}

Its all there including the two new lines. Here's code Removing your P2 function and reading/printing the byte values of the ascii.

#include <SPI.h>
#include <SD.h>

void p1(char string[], int val) {
  Serial.print(string);
  Serial.print(":");
  Serial.println(val);
}
//void p2(char string[], char a){Serial.print(string);Serial.write(a);Serial.println();}

File file;

void setup() {
  Serial.begin(9600);
  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }

  //Count the number of characters.
  file = SD.open("Test_1.txt", FILE_READ);
  if (file) {
    byte newChar;
    int counter = 0;
    file.seek(0);
    while (file.available()) {
      newChar = file.read();
      Serial.print ("newChar = ");
      Serial.print(newChar);
      Serial.print('\t');
      if(newChar==10){
      Serial.println("new line");
      }
      else
      Serial.println((char)newChar);
      counter++;
      p1("Counter", counter);
    }
  }


  else Serial.println("Could not open file for reading.");
  file.close();

  Serial.println("Done.");
}

void loop() {
}

cattledog:
Its all there including the two new lines. Here's code Removing your P2 function and reading/printing the byte values of the ascii.

#include <SPI.h>

#include <SD.h>

void p1(char string[], int val) {
  Serial.print(string);
  Serial.print(":");
  Serial.println(val);
}
//void p2(char string[], char a){Serial.print(string);Serial.write(a);Serial.println();}

File file;

void setup() {
  Serial.begin(9600);
  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }

//Count the number of characters.
  file = SD.open("Test_1.txt", FILE_READ);
  if (file) {
    byte newChar;
    int counter = 0;
    file.seek(0);
    while (file.available()) {
      newChar = file.read();
      Serial.print ("newChar = ");
      Serial.print(newChar);
      Serial.print('\t');
      if(newChar==10){
      Serial.println("new line");
      }
      else
      Serial.println((char)newChar);
      counter++;
      p1("Counter", counter);
    }
  }

else Serial.println("Could not open file for reading.");
  file.close();

Serial.println("Done.");
}

void loop() {
}

Hi,

Thanks for the response. When I hit "enter" in a .txt file, how many characters are inserted? What are they?

I though only 1 character was inserted (newline). Your code tells me otherwise. It says that ascii character 13 (Carriage Return) and 10 (newline) are inserted for each "newline".

EDITED:

This is an infuriating point of fact. Thank you for finding out the mysterious character, cattledog.

Thank you for finding out the mysterious character, cattledog.

I don't want to be picky, but did you read what I said in reply #6 ?

After the alphabet and the numbers there are two Carriage Return/Linefeed pairs in the file.

UKHeliBob:
I don't want to be picky, but did you read what I said in reply #6 ?

Yes, I did. Unfortunately, I didn't understand it at the time (since I didn't have the code nor the article to support your point). In hindsight, everything you wrote was spot on!

Thanks!

But to recap: The issue was that I presumed there was 1 character (just the \n) to mark the end of a line when in fact a \r\n pair marked the end of a line.

In Word, if you do a file/save as a plan text file you get the choice of what the line ending should be