[Solved] Discrepancy between file.available() and file.size() on larger files.

[blake3]

Hi. I have a 591 kb text file of random sentences, delimited by a special character. I plan on reading text out of this file, one byte at a time. In doing some testing for this project, and to learn how to use the SD class, I have the following code.

I have noticed that file.available() and file.size() return different values (file available tops out at 32767, the max int value, while file size goes to 591272, the actual file size).

Is it a problem that available and size do not give the same value? I had planned on using the value of available to find my way around the file as I read through it.

If I newly open the 591 kb file and ask for bytes available(), wouldn't the actual number of bytes available be more than the value the command can return? Or am I missing something conceptually?

Why doesn't file.available() return an unsigned long?

Thanks,
Blake

Code: [Select]

#include <SD.h>
#include <SPI.h>

File myFile1;

const int chipSelect = BUILTIN_SDCARD; //built in sd card
boolean SDfound;

int totalBytesA = 0;
long totalBytesS = 0;

//long digit1 = 500400300;

void setup() {

  Serial.begin(38400);
  delay(3000);

  if (SDfound == 0) {  // make sure SD card is available
    if (!SD.begin(chipSelect)) {
      Serial.print("not found");
      while (1);
    }
  }
  SDfound = 1;

  myFile1 = SD.open("words.txt");
  if (myFile1) {
    
    Serial.print("words.txt, total bytes, size: ");
    totalBytesS = myFile1.size();
    Serial.println(totalBytesS);
    
    Serial.print("words.txt, total bytes, avail: ");
    totalBytesA = myFile1.available();
    Serial.println(totalBytesA);
    
    // close the file:
    myFile1.close();

  } else {
    // if the file didn't open, print an error:
    Serial.println("error opening words.txt");
  }

}

void loop() {
//nothing
}

Output:

Code: [Select]


words.txt, total bytes, size: 591272
words.txt, total bytes, avail: 32767


[blake3]

Trying to answer my own question here, in this forum post BulldogLowell at first implies that file.available() only returns the amount of characters in the Serial buffer:

https://forum.arduino.cc/index.php?topic=530700.0

there is a Serial buffer, containing room for a fixed number of characters.

the bytes available in that buffer is decremented as chars are removed by receiver.

...

available() returns true as long as there are any characters in the buffer.

PaulS then appears to contradict this, saying:

"there is a Serial buffer, containing room for a fixed number of characters. the bytes available in that buffer is decremented as chars are removed by receiver."

While that is true, that has nothing to do with reading from a file.

The available() method you are using tells you how many bytes between where you last read from the file and the end of the file.

Maybe it only reads to the end on files smaller than 32767 bytes, because the file is smaller than some buffer? On files larger than 32767 bytes, available() returns the buffer size?

Also, maybe this is ridiculous, but just want to add that I tried changing totalBytesA to a long variable type, which predictably, didn't work.

[blake3]

Solved (or at least better understood):

This forum post describes the problem in detail, and that Robtillaart reported it as an issue.
https://forum.arduino.cc/index.php?topic=66929.0

Using size() - position() instead of available() is perhaps the way to go.

The issue report (https://code.google.com/archive/p/arduino/issues/571) also indicates several fixes that involve modifying file.cpp

[bhupiister]

Hey Blake, thanks for the explanation.I know its too old post but still trying my luck to contact you. I am going through a similar problem. I have a filed in SD card which i want to read and upload it on the web server over 2G modem SIM800L via ESP32. I am facing similar problem, here is my code.

Code: [Select]

if (File1) {
int i = 0;
while (File1.available()>0) {
  char_buffer = File1.read();//Takes one by one character in char_buffer from File.read()
  Serial.println("char_buffer: " + String(char_buffer));
  string_buffer.concat(char_buffer);//Concats every single character from char_buffer and makes string_buffer
  Serial.println("string_buffer: " + String(string_buffer));
  i++;
  if (i == buffer_space) {
    sendATcommand("AT+FTPPUT=2," + String(buffer_space), "AT+FTPPUT=2,10", "ERROR", 1000);
    sendATcommand(string_buffer, "OK", "ERROR", 5000);
    string_buffer = "";
    i = 0;
  }
}
if (string_buffer != ""){
  sendATcommand("AT+FTPPUT=2," + String(i), "AT+FTPPUT=2,10", "ERROR", 1000);
  sendATcommand(string_buffer, "OK", "ERROR", 5000);
  sendATcommand("AT+FTPPUT=2,0", "OK", "ERROR", 1000);
}
File1.close();
}

If my file size is small, i have a successful upload, but if i have a large file, the loop keeps going on and uploads more than the file size, i have to stop the operation. If i stop and read the file on server, it has garbage text at the end.

I guess its because of the problem you just defined. Can you let me know how can i solve it in this code?