Pages: [1] 2   Go Down
Author Topic: SD Card Directory Listing Fails within Large Application  (Read 2149 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Over recent months I have been building a home automation system. With over 3,500 lines of code I have solved most of my software requirements including a web based user interface and the use of SD cards for data storage.

The program compiles to about 90K and after using flash strings and putting another 1.5K or strings into EEPROM the application starts up and runs with about 2K of free RAM. I am using a Freetronics Ethermega card (256MB Flash, 8KB RAM)

I am having trouble with attempts to reliably list all the files on my SD card. It seems to work OK for up to 15 files but when there are more files are on the card some of them do not appear in directory listings using openNextFile() and in some cases none of them appear.

The following code works just fine when complied and run on my EtherMega card when there are 16 files on the card. I get the correct list of files and file count in the serial monitor.

Code:
#include <SD.h>
//------------------------------------------------------------
void setup() {

  //initialise serial port
  Serial.begin(9600);
  Serial.println("Setup");

  //disable hardware SS pin
  pinMode(53,OUTPUT); 
  digitalWrite(53,HIGH);
  //disable Ethernet
  pinMode(10,OUTPUT); 
  digitalWrite(10,HIGH); //Ethernet
 
  //initialise SD
  pinMode(4,OUTPUT); 
  digitalWrite(4,LOW);
  SD.begin(4);

  Serial.println("SD Card File List");
  File root = SD.open("/");
  printDirectory(root);
  root.close();
  Serial.println("done!"); 
 
}
//------------------------------------------------------------
void loop () {
  //do nothing 
}
//------------------------------------------------------------
void printDirectory(File dir) {
 
   dir.rewindDirectory();
   int l_count = 0;
   while(true) {
     File entry =  dir.openNextFile();
     if (! entry) {
       break;
     }
     if (!entry.isDirectory()) {
       l_count++;
       Serial.println(entry.name());
     }
     entry.close();
   }
   Serial.println(String("File Count is " + String(l_count)));
}
//------------------------------------------------------------
//Serial Monitor Listing
//------------------------------------------------------------
Setup
SD Card File List
11021200.TXT
11021100.TXT
HTML1102.TXT
11021300.TXT
ACTV1030.TXT
ACTV1031.TXT
ACTV1101.TXT
HACK1031.TXT
HACK1101.TXT
HIST2013.TXT
HTML1030.TXT
HTML1031.TXT
HTML1101.TXT
ACTV1102.TXT
HACK1102.TXT
11021400.TXT <<This file dissappears when printDirectory() is run within the main application
File Count is 16
done!
//------------------------------------------------------------

But when I put the printDirectory() procedure in my main application and call it at the end of the setup() procedure (as above) I can only get a list of 15 files in the serial monitor. The last (16th) file created is missing from the list. I get the same result within the main application that lists the files (with file sizes) on an application web page.

Has anyone got any suggestions? I have chased this issue for two weeks without success. I am wondering if perhaps I need to format the SD card in a certain way - but why do all of the other SD card operations (writing and reading) work OK?

I am using a 1GB AData SD card formatted with allocation units of 16KB and about 61,000 sectors. (I think I have that described correctly.)

All my testings, including free RAM measurement throughout my software indicates that I am not running out of RAM.

Thanks for your help.

Catweazle, New Zealand
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Here is a small update:

My web site is currently listing these sixteen files:

Code:
LIST OF FILES
11021100.TXT 582
11021600.TXT 590
11021700.TXT 590
11021800.TXT 590
ACTV1030.TXT 13316
ACTV1031.TXT 10873
ACTV1101.TXT 13250
ACTV1102.TXT 17396
HACK1031.TXT 695
HACK1101.TXT 441
HACK1102.TXT 1602
HIST2013.TXT 7321
HTML1030.TXT 11365
HTML1031.TXT 11436
HTML1101.TXT 8491
HTML1102.TXT 8826
16 files
But I know that there are at least 17 files on the SD card because it is 7:19PM and the hourly backup file "11021900.TXT" has been created. Via another function I can open and browse the "11021900.TXT" so I am certain it is there - it is just not shown in the root directory listing.

As I get more and more files on the card I expect more than one file not to display and I have seen instances where there are about 25 files on the SD card (as seen in MS Windows), no files appears in my web site's SD card root file directory list and yet, via other arduino web site functionality functionality I can open and display any of them.

Once again it seems that the last file created on my SD card is not showing in the directory listing because the SD card openNextFile() function is not giving it up.

Cheers

Catweazle NZ
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Now after 8:00pm and this is displayed:

Code:
LIST OF FILES
11021100.TXT 582
11021700.TXT 590
11021800.TXT 590
ACTV1030.TXT 13316
ACTV1031.TXT 10873
ACTV1101.TXT 13250
ACTV1102.TXT 17638
HACK1031.TXT 695
HACK1101.TXT 441
HACK1102.TXT 1602
HIST2013.TXT 7321
HTML1030.TXT 11365
HTML1031.TXT 11436
HTML1101.TXT 8491
HTML1102.TXT 10157
15 files

We are back to 15 files.

The file "11021600.TXT" has been deleted by the system - hence no display. But the 11021900.TXT and 1102200.TXT hourly backup files are on the SD Card and can be opened and viewed using other functionality. But these last two created files do not appear in the SD card root directory file listing that relies on openNextFile().

I can only presume that there is a bug in the SD card functionality that walks through the file entries in the SD card's directory structure pages. Perhaps the 15/16th files are in the second 512KB block that it reads and somehow there is a bug in moving from the 1st to the second 512KB data block.

Cheers

Catweazle NZ
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 549
Posts: 46090
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
   Serial.println(String("File Count is " + String(l_count)));
You have a limited amount of memory. You know that. And, yet, you piss away resources on Strings, so that you can avoid having two Serial.print() statements. Why?

Is the real code littered with crap like this, too?
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
   Serial.println(String("File Count is " + String(l_count)));
You have a limited amount of memory. You know that. And, yet, you piss away resources on Strings, so that you can avoid having two Serial.print() statements. Why?

Is the real code littered with crap like this, too?
I have plenty of Flash memory available and almost 2K or free RAM at application startup after using F() strings extensively and moving 1.5K of strings to EEPROM. Continuous analysis thoughout my software suggests that free RAM is never less than  1K.

07:00:00 MINIMUM FREE RAM
- Procedure FileMMDDWrite
- Time 06:14:53
- Min Free RAM 1441
08:00:00 MINIMUM FREE RAM
- Procedure WebServerProcess
- Time 07:00:00
- Min Free RAM 1817
09:00:00 MINIMUM FREE RAM
- Procedure SDCardSortedFileList <<This is the program that is the subject of this thread topic
- Time 08:01:45
- Min Free RAM 1314                      << Unless the SD library needs more than 1300 bytes it will not run out of RAM

And don''t forget I am still building my application. I can dot every I and cross every T later on thank you very much.

Cheers

Catweazle NZ
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 549
Posts: 46090
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I can dot every I and cross every T later on thank you very much.
OK, fine. I, on the other hand, prefer to dot my i's and cross my t's as I write them, rather than later.
Logged

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1473
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

There is a serious bug in the Arduino printDirectory() example and you have it in your version.

The File object is a complex class with a pointer to a chunk of dynamic memory.  You must not call functions using call by value for arguments that are File objects.

This form causes a copy of the file object to be made in calls  and results in two pointers to the same dynamic memory area.
Code:
void printDirectory(File dir)

You should either use a pointer or call by reference.
Code:
void printDirectory(File& dir)
// or
void printDirectory(File* dir)

The SD.h library is very fragile and has a number of bugs so this may also be part of your problem.

SD.h is a wrapper for an old version of SdFat that I wrote in 2009.   The SdFat base code has not been maintained.

I suggest that you isolate the bug and post code and the procedure to demonstrate the bug so we can help.
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

There is a serious bug in the Arduino printDirectory() example and you have it in your version.

The File object is a complex class with a pointer to a chunk of dynamic memory.  You must not call functions using call by value for arguments that are File objects.

This form causes a copy of the file object to be made in calls  and results in two pointers to the same dynamic memory area.
Code:
void printDirectory(File dir)

You should either use a pointer or call by reference.
Code:
void printDirectory(File& dir)
// or
void printDirectory(File* dir)

The SD.h library is very fragile and has a number of bugs so this may also be part of your problem.

SD.h is a wrapper for an old version of SdFat that I wrote in 2009.   The SdFat base code has not been maintained.

I suggest that you isolate the bug and post code and the procedure to demonstrate the bug so we can help.
Thanks for this advice. It is a shame that the issues exists in standard SD library examples and encourages everybody to embed the error into their code.

I have made the change in my software and it still runs. However I only have 14 files on the SD card at the moment and the problem does not manifest, presumably because all the file directory entries exist within the first 512KB data block. I will put some more files back on the card tonight (+ 14 hours) and then restest to see if my directory listing procedure is able to extract the details of all file entries in the second 512KB data block. (Assuming that is the problem.)

If the problem still exists then I will launch a debugging process through the SD and underlying sdfat libraries to see if I can identify the problem.

Since the problem does not exist on my card when I use a minimalist directory listing program but does exist when I embed the directory listing procedure within my 90KB application then I suspect it is free memory related. Within my application I have about 2K or free RAM and down to about 1300 bytes (because of stack/heap growth) when I enter the directory listing procedure. I would have thought that would be enough for the SD/SDFat libraries to work OK.

I can also ramp up the RAM usage in my simple directory listing test program to see I can get it to fail when free RAM gets down to about the same as my main application.

I am surprised that the SD/SDFat libraries have not been well maintained over recent years. Data storage is an obvious core software application requirement - I would have expected many others to have found this problem in the past and for it to have long since been resolved.

Thanks for the assistance. Stand by for more updates.

Catweazle NZ
Logged

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1473
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I have about 2K or free RAM and down to about 1300 bytes (because of stack/heap growth) when I enter the directory listing procedure. I would have thought that would be enough for the SD/SDFat libraries to work OK.
This amount of RAM should be more than adequate.

Quote
I am surprised that the SD/SDFat libraries have not been well maintained over recent years. Data storage is an obvious core software application requirement - I would have expected many others to have found this problem in the past and for it to have long since been resolved.

Unfortunately The Arduino company didn't contact me when they wrote the SD.h wrapper.  I was doing a total rewrite of SdFat and now it is not easy for them to move to the current version of SdFat.

It would be unfortunate if they did move to the current SdFat since I have now written a much more general FAT12/FAT16/FAT32 library.   I plan to support USB flash drives and any other block device, even USB hard drives.
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I have about 2K or free RAM and down to about 1300 bytes (because of stack/heap growth) when I enter the directory listing procedure. I would have thought that would be enough for the SD/SDFat libraries to work OK.
This amount of RAM should be more than adequate.

Quote
I am surprised that the SD/SDFat libraries have not been well maintained over recent years. Data storage is an obvious core software application requirement - I would have expected many others to have found this problem in the past and for it to have long since been resolved.

Unfortunately The Arduino company didn't contact me when they wrote the SD.h wrapper.  I was doing a total rewrite of SdFat and now it is not easy for them to move to the current version of SdFat.

It would be unfortunate if they did move to the current SdFat since I have now written a much more general FAT12/FAT16/FAT32 library.   I plan to support USB flash drives and any other block device, even USB hard drives.
Your situation/view point sounds just like a typical large organization/bureaucracy just not being able to accommodate the talents, enthusiasm and needs of their brightest staff. And the organization and their end-customers always suffers because of it.

Thank you so much for your assistance so far with my problem. I hope to be able to resume my testing in a couple of hours.

Catweazle NZ
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

No change. When I put 30 files on the card directory listings work fine within the test sketch I listed at the start of this topic. But within my large application the printDirectory() procedure when called from setup() lists just 15 files.

I have a separate sorted directory listing procedure within my application. For some reason it is able to list 18 files - but it is parsing the SD card directory multiple times which suggests that after rewindDirectory it gets further through the entries some of the time.
Code:
void printDirectory(File dir),
void printDirectory(File& dir) and
void printDirectory(File* dir)

All seem to work OK.

Nothing for it but some detailed analysis of the SD and SDFat source files and a liberal use of Serial.println to see if I can isolate the problem.

Maybe next weekend.

Catweazle NZ
Logged

0
Offline Offline
Edison Member
*
Karma: 44
Posts: 1473
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
No change. When I put 30 files on the card directory listings work fine within the test sketch I listed at the start of this topic. But within my large application the printDirectory() procedure when called from setup() lists just 15 files.
Do you call printDirectory() before any other setup code ?  Just initialize the SD and call printDirectory().

This sounds like a common problem I have seen for both SdFat and SD.h.  Often an application writes over internal memory in SD libraries and the result appears to be a bug in the SD library.  I have seen this at least a dozen times.  Check all array sizes and make sure all array indices are correct.

It is possible for memory to be corrupted before setup() if you have classes with constructors that are called at upstart.

Interrupt routines sometimes are the cause.

Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
No change. When I put 30 files on the card directory listings work fine within the test sketch I listed at the start of this topic. But within my large application the printDirectory() procedure when called from setup() lists just 15 files.
Do you call printDirectory() before any other setup code ?  Just initialize the SD and call printDirectory().

This sounds like a common problem I have seen for both SdFat and SD.h.  Often an application writes over internal memory in SD libraries and the result appears to be a bug in the SD library.  I have seen this at least a dozen times.  Check all array sizes and make sure all array indices are correct.

It is possible for memory to be corrupted before setup() if you have classes with constructors that are called at upstart.

Interrupt routines sometimes are the cause.
Thanks for the suggestion - I am moving forward now.

A call to printDirectory() at the start of my setup procedure lists 27 files - which I think is what is on the card right now after I deleted a couple of files last night.

A second call to printDirectory() at the end of my setup procedure lists 15 files!

So tonight (+ 14 hours) I will be able to move the printDirectory() calls up and down my setup procedure to find out where the problem is.

Yes there are arrays in there but my application seems to have been working well and as expected for several weeks (except for this issue). That said there has been the very occasional unexpected action so it is possible that I have some memory corruption related to out of bounds array operations.

Standby for another update. Thank you so much for pointing me in what may be the right direction.

Catweazle NZ
Logged

Offline Offline
Jr. Member
**
Karma: 2
Posts: 79
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Problem Solved!

First of all I tidied up the directory listing test code in my main application from this:

Code:
void setup() {
  //SD Card initialisation
  Serial.println("Setup Start");
  File root = SD.open("/");
  printDirectory(root);
  root.close();
  Serial.println("done!");
 
  //Other setup code in here

  Serial.println("Setup end");
  File root = SD.open("/");
  printDirectory(root);
  root.close();
  Serial.println("done!");
 } 

void printDirectory(File & dir) {
 
   dir.rewindDirectory();
   int l_count = 0;
   while(true) {
     File entry =  dir.openNextFile();
     if (! entry) {
       break;
     }
     if (!entry.isDirectory()) {
       l_count++;
       Serial.println(entry.name());
     }
     entry.close();
   }
   Serial.println(String("File Count is " + String(l_count)));
}
The above test code in my application this morning extracted 27 files on the first call to printDirectory() and 15 files on the second call - which suggested some problem in between.

To this:
Code:
void setup() {

  //SD Card initialisation
  printDirectory("Setup Start");
 
  //Other setup code in here
  printDirectory("Before Ethernet");

  //Other setup code in here
  printDirectory("After Ethernet");

  //Other setup code in here
  printDirectory("After NTP UDP");

  //Other setup code in here
  printDirectory("After Climate Array Init");

  //Other setup code in here
  printDirectory("After Startup Email");
  printDirectory("Setup End");
 } 

void printDirectory(String p_locn) {

  Serial.println(p_locn);
  SPIDeviceSelect(DC_SDCardSSPin);

  File l_root = SD.open("/");
  l_root.rewindDirectory();
  int l_count = 0;
  while(true) {
    File l_entry =  l_root.openNextFile();
    if (! l_entry) {
      break;
    }
    if (!l_entry.isDirectory()) {
      l_count++;
      //Serial.println(l_entry.name());
    }
    l_entry.close();
  }
  l_root.close();
  Serial.print(F("File Count is "));
  Serial.println(l_count);

  Serial.println(F("done!"));
  SPIDeviceSelect(DC_EthernetSSPin);
}

and I got 27 files listed for every call to printDirectory(). Nothing in my application setup now caused the list (count) of files in the root directory to get 15 files only. But my web based directory listing still only showed fifteen files and was using the old convention of passing the root directory object to the directory listing procedure.

So I moved the root directory object file creation, use and close to the actual web sorted directory listing procedure in my application and it now lists all the files in the root directory of my SD card.

Which means (I think) that there may still be a bug in the SD library related to where the root directory file object is created. Basically it needs to be in the procedure where you need it and it cannot be passed as an argument to another procedure (except maybe once - but not in subsequent invocations.)

Well it is all working for me now. Thanks to fat16lib for a testing suggestion which eventually led to my discovery of the solution. I would be interested to hear from others as to whether or not they can replicate this problem - if they can then hopefully someone will possibly resolve the underlying bug/problem.

Cheers for now.

Catweazle NZ
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 549
Posts: 46090
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Which means (I think) that there may still be a bug in the SD library related to where the root directory file object is created. Basically it needs to be in the procedure where you need it and it cannot be passed as an argument to another procedure (except maybe once - but not in subsequent invocations.)
You need to post some code that illustrates this assertion.
Logged

Pages: [1] 2   Go Up
Jump to: