Over recent months I have been building a home automation system. With over 3,500 lines of code I have solved most of my software requirements including a web based user interface and the use of SD cards for data storage.
The program compiles to about 90K and after using flash strings and putting another 1.5K or strings into EEPROM the application starts up and runs with about 2K of free RAM. I am using a Freetronics Ethermega card (256MB Flash, 8KB RAM)
I am having trouble with attempts to reliably list all the files on my SD card. It seems to work OK for up to 15 files but when there are more files are on the card some of them do not appear in directory listings using openNextFile() and in some cases none of them appear.
The following code works just fine when complied and run on my EtherMega card when there are 16 files on the card. I get the correct list of files and file count in the serial monitor.
#include <SD.h>
//------------------------------------------------------------
void setup() {
//initialise serial port
Serial.begin(9600);
Serial.println("Setup");
//disable hardware SS pin
pinMode(53,OUTPUT);
digitalWrite(53,HIGH);
//disable Ethernet
pinMode(10,OUTPUT);
digitalWrite(10,HIGH); //Ethernet
//initialise SD
pinMode(4,OUTPUT);
digitalWrite(4,LOW);
SD.begin(4);
Serial.println("SD Card File List");
File root = SD.open("/");
printDirectory(root);
root.close();
Serial.println("done!");
}
//------------------------------------------------------------
void loop () {
//do nothing
}
//------------------------------------------------------------
void printDirectory(File dir) {
dir.rewindDirectory();
int l_count = 0;
while(true) {
File entry = dir.openNextFile();
if (! entry) {
break;
}
if (!entry.isDirectory()) {
l_count++;
Serial.println(entry.name());
}
entry.close();
}
Serial.println(String("File Count is " + String(l_count)));
}
//------------------------------------------------------------
//Serial Monitor Listing
//------------------------------------------------------------
Setup
SD Card File List
11021200.TXT
11021100.TXT
HTML1102.TXT
11021300.TXT
ACTV1030.TXT
ACTV1031.TXT
ACTV1101.TXT
HACK1031.TXT
HACK1101.TXT
HIST2013.TXT
HTML1030.TXT
HTML1031.TXT
HTML1101.TXT
ACTV1102.TXT
HACK1102.TXT
11021400.TXT <<This file dissappears when printDirectory() is run within the main application
File Count is 16
done!
//------------------------------------------------------------
But when I put the printDirectory() procedure in my main application and call it at the end of the setup() procedure (as above) I can only get a list of 15 files in the serial monitor. The last (16th) file created is missing from the list. I get the same result within the main application that lists the files (with file sizes) on an application web page.
Has anyone got any suggestions? I have chased this issue for two weeks without success. I am wondering if perhaps I need to format the SD card in a certain way - but why do all of the other SD card operations (writing and reading) work OK?
I am using a 1GB AData SD card formatted with allocation units of 16KB and about 61,000 sectors. (I think I have that described correctly.)
All my testings, including free RAM measurement throughout my software indicates that I am not running out of RAM.
But I know that there are at least 17 files on the SD card because it is 7:19PM and the hourly backup file "11021900.TXT" has been created. Via another function I can open and browse the "11021900.TXT" so I am certain it is there - it is just not shown in the root directory listing.
As I get more and more files on the card I expect more than one file not to display and I have seen instances where there are about 25 files on the SD card (as seen in MS Windows), no files appears in my web site's SD card root file directory list and yet, via other arduino web site functionality functionality I can open and display any of them.
Once again it seems that the last file created on my SD card is not showing in the directory listing because the SD card openNextFile() function is not giving it up.
The file "11021600.TXT" has been deleted by the system - hence no display. But the 11021900.TXT and 1102200.TXT hourly backup files are on the SD Card and can be opened and viewed using other functionality. But these last two created files do not appear in the SD card root directory file listing that relies on openNextFile().
I can only presume that there is a bug in the SD card functionality that walks through the file entries in the SD card's directory structure pages. Perhaps the 15/16th files are in the second 512KB block that it reads and somehow there is a bug in moving from the 1st to the second 512KB data block.
Serial.println(String("File Count is " + String(l_count)));
You have a limited amount of memory. You know that. And, yet, you piss away resources on Strings, so that you can avoid having two Serial.print() statements. Why?
Is the real code littered with crap like this, too?
Serial.println(String("File Count is " + String(l_count)));
You have a limited amount of memory. You know that. And, yet, you piss away resources on Strings, so that you can avoid having two Serial.print() statements. Why?
Is the real code littered with crap like this, too?
I have plenty of Flash memory available and almost 2K or free RAM at application startup after using F() strings extensively and moving 1.5K of strings to EEPROM. Continuous analysis thoughout my software suggests that free RAM is never less than 1K.
07:00:00 MINIMUM FREE RAM
Procedure FileMMDDWrite
Time 06:14:53
Min Free RAM 1441
08:00:00 MINIMUM FREE RAM
Procedure WebServerProcess
Time 07:00:00
Min Free RAM 1817
09:00:00 MINIMUM FREE RAM
Procedure SDCardSortedFileList <<This is the program that is the subject of this thread topic
Time 08:01:45
Min Free RAM 1314 << Unless the SD library needs more than 1300 bytes it will not run out of RAM
And don''t forget I am still building my application. I can dot every I and cross every T later on thank you very much.
There is a serious bug in the Arduino printDirectory() example and you have it in your version.
The File object is a complex class with a pointer to a chunk of dynamic memory. You must not call functions using call by value for arguments that are File objects.
This form causes a copy of the file object to be made in calls and results in two pointers to the same dynamic memory area.
void printDirectory(File dir)
You should either use a pointer or call by reference.
void printDirectory(File& dir)
// or
void printDirectory(File* dir)
The SD.h library is very fragile and has a number of bugs so this may also be part of your problem.
SD.h is a wrapper for an old version of SdFat that I wrote in 2009. The SdFat base code has not been maintained.
I suggest that you isolate the bug and post code and the procedure to demonstrate the bug so we can help.
fat16lib:
There is a serious bug in the Arduino printDirectory() example and you have it in your version.
The File object is a complex class with a pointer to a chunk of dynamic memory. You must not call functions using call by value for arguments that are File objects.
This form causes a copy of the file object to be made in calls and results in two pointers to the same dynamic memory area.
void printDirectory(File dir)
You should either use a pointer or call by reference.
void printDirectory(File& dir)
// or
void printDirectory(File* dir)
The SD.h library is very fragile and has a number of bugs so this may also be part of your problem.
SD.h is a wrapper for an old version of SdFat that I wrote in 2009. The SdFat base code has not been maintained.
I suggest that you isolate the bug and post code and the procedure to demonstrate the bug so we can help.
Thanks for this advice. It is a shame that the issues exists in standard SD library examples and encourages everybody to embed the error into their code.
I have made the change in my software and it still runs. However I only have 14 files on the SD card at the moment and the problem does not manifest, presumably because all the file directory entries exist within the first 512KB data block. I will put some more files back on the card tonight (+ 14 hours) and then restest to see if my directory listing procedure is able to extract the details of all file entries in the second 512KB data block. (Assuming that is the problem.)
If the problem still exists then I will launch a debugging process through the SD and underlying sdfat libraries to see if I can identify the problem.
Since the problem does not exist on my card when I use a minimalist directory listing program but does exist when I embed the directory listing procedure within my 90KB application then I suspect it is free memory related. Within my application I have about 2K or free RAM and down to about 1300 bytes (because of stack/heap growth) when I enter the directory listing procedure. I would have thought that would be enough for the SD/SDFat libraries to work OK.
I can also ramp up the RAM usage in my simple directory listing test program to see I can get it to fail when free RAM gets down to about the same as my main application.
I am surprised that the SD/SDFat libraries have not been well maintained over recent years. Data storage is an obvious core software application requirement - I would have expected many others to have found this problem in the past and for it to have long since been resolved.
Thanks for the assistance. Stand by for more updates.
I have about 2K or free RAM and down to about 1300 bytes (because of stack/heap growth) when I enter the directory listing procedure. I would have thought that would be enough for the SD/SDFat libraries to work OK.
This amount of RAM should be more than adequate.
I am surprised that the SD/SDFat libraries have not been well maintained over recent years. Data storage is an obvious core software application requirement - I would have expected many others to have found this problem in the past and for it to have long since been resolved.
Unfortunately The Arduino company didn't contact me when they wrote the SD.h wrapper. I was doing a total rewrite of SdFat and now it is not easy for them to move to the current version of SdFat.
It would be unfortunate if they did move to the current SdFat since I have now written a much more general FAT12/FAT16/FAT32 library. I plan to support USB flash drives and any other block device, even USB hard drives.
I have about 2K or free RAM and down to about 1300 bytes (because of stack/heap growth) when I enter the directory listing procedure. I would have thought that would be enough for the SD/SDFat libraries to work OK.
This amount of RAM should be more than adequate.
I am surprised that the SD/SDFat libraries have not been well maintained over recent years. Data storage is an obvious core software application requirement - I would have expected many others to have found this problem in the past and for it to have long since been resolved.
Unfortunately The Arduino company didn't contact me when they wrote the SD.h wrapper. I was doing a total rewrite of SdFat and now it is not easy for them to move to the current version of SdFat.
It would be unfortunate if they did move to the current SdFat since I have now written a much more general FAT12/FAT16/FAT32 library. I plan to support USB flash drives and any other block device, even USB hard drives.
Your situation/view point sounds just like a typical large organization/bureaucracy just not being able to accommodate the talents, enthusiasm and needs of their brightest staff. And the organization and their end-customers always suffers because of it.
Thank you so much for your assistance so far with my problem. I hope to be able to resume my testing in a couple of hours.
No change. When I put 30 files on the card directory listings work fine within the test sketch I listed at the start of this topic. But within my large application the printDirectory() procedure when called from setup() lists just 15 files.
I have a separate sorted directory listing procedure within my application. For some reason it is able to list 18 files - but it is parsing the SD card directory multiple times which suggests that after rewindDirectory it gets further through the entries some of the time.
void printDirectory(File dir),
void printDirectory(File& dir) and
void printDirectory(File* dir)
All seem to work OK.
Nothing for it but some detailed analysis of the SD and SDFat source files and a liberal use of Serial.println to see if I can isolate the problem.
No change. When I put 30 files on the card directory listings work fine within the test sketch I listed at the start of this topic. But within my large application the printDirectory() procedure when called from setup() lists just 15 files.
Do you call printDirectory() before any other setup code ? Just initialize the SD and call printDirectory().
This sounds like a common problem I have seen for both SdFat and SD.h. Often an application writes over internal memory in SD libraries and the result appears to be a bug in the SD library. I have seen this at least a dozen times. Check all array sizes and make sure all array indices are correct.
It is possible for memory to be corrupted before setup() if you have classes with constructors that are called at upstart.
No change. When I put 30 files on the card directory listings work fine within the test sketch I listed at the start of this topic. But within my large application the printDirectory() procedure when called from setup() lists just 15 files.
Do you call printDirectory() before any other setup code ? Just initialize the SD and call printDirectory().
This sounds like a common problem I have seen for both SdFat and SD.h. Often an application writes over internal memory in SD libraries and the result appears to be a bug in the SD library. I have seen this at least a dozen times. Check all array sizes and make sure all array indices are correct.
It is possible for memory to be corrupted before setup() if you have classes with constructors that are called at upstart.
Interrupt routines sometimes are the cause.
Thanks for the suggestion - I am moving forward now.
A call to printDirectory() at the start of my setup procedure lists 27 files - which I think is what is on the card right now after I deleted a couple of files last night.
A second call to printDirectory() at the end of my setup procedure lists 15 files!
So tonight (+ 14 hours) I will be able to move the printDirectory() calls up and down my setup procedure to find out where the problem is.
Yes there are arrays in there but my application seems to have been working well and as expected for several weeks (except for this issue). That said there has been the very occasional unexpected action so it is possible that I have some memory corruption related to out of bounds array operations.
Standby for another update. Thank you so much for pointing me in what may be the right direction.
The above test code in my application this morning extracted 27 files on the first call to printDirectory() and 15 files on the second call - which suggested some problem in between.
To this:
void setup() {
//SD Card initialisation
printDirectory("Setup Start");
//Other setup code in here
printDirectory("Before Ethernet");
//Other setup code in here
printDirectory("After Ethernet");
//Other setup code in here
printDirectory("After NTP UDP");
//Other setup code in here
printDirectory("After Climate Array Init");
//Other setup code in here
printDirectory("After Startup Email");
printDirectory("Setup End");
}
void printDirectory(String p_locn) {
Serial.println(p_locn);
SPIDeviceSelect(DC_SDCardSSPin);
File l_root = SD.open("/");
l_root.rewindDirectory();
int l_count = 0;
while(true) {
File l_entry = l_root.openNextFile();
if (! l_entry) {
break;
}
if (!l_entry.isDirectory()) {
l_count++;
//Serial.println(l_entry.name());
}
l_entry.close();
}
l_root.close();
Serial.print(F("File Count is "));
Serial.println(l_count);
Serial.println(F("done!"));
SPIDeviceSelect(DC_EthernetSSPin);
}
and I got 27 files listed for every call to printDirectory(). Nothing in my application setup now caused the list (count) of files in the root directory to get 15 files only. But my web based directory listing still only showed fifteen files and was using the old convention of passing the root directory object to the directory listing procedure.
So I moved the root directory object file creation, use and close to the actual web sorted directory listing procedure in my application and it now lists all the files in the root directory of my SD card.
Which means (I think) that there may still be a bug in the SD library related to where the root directory file object is created. Basically it needs to be in the procedure where you need it and it cannot be passed as an argument to another procedure (except maybe once - but not in subsequent invocations.)
Well it is all working for me now. Thanks to fat16lib for a testing suggestion which eventually led to my discovery of the solution. I would be interested to hear from others as to whether or not they can replicate this problem - if they can then hopefully someone will possibly resolve the underlying bug/problem.
Which means (I think) that there may still be a bug in the SD library related to where the root directory file object is created. Basically it needs to be in the procedure where you need it and it cannot be passed as an argument to another procedure (except maybe once - but not in subsequent invocations.)
You need to post some code that illustrates this assertion.
Which means (I think) that there may still be a bug in the SD library related to where the root directory file object is created.
Not likely. You just are avoiding the real problem by getting a fresh copy of the root file object. when you open root, SD.h just gives you a copy of its internal root object.
Which means (I think) that there may still be a bug in the SD library related to where the root directory file object is created.
Not likely. You just are avoiding the real problem by getting a fresh copy of the root file object. when you open root, SD.h just gives you a copy of its internal root object.
I will try to do some more testing on the weekend.
At this stage I seem to have posted SD directory reading code that works and code that does not work within my application when everything else remains unchanged. I accept that the logic of my code suggests both types of code should generate the same result - but they do not but I guess my code cannot be corrupting an SD root directory file object if it is not permanently instantiated while the application is off doing other things. Perhaps my application in those circumstances is corrupting other things that I have not yet discovered.
Well I finally got to the bottom of my problem thanks to a few suggestions from fat16lib that forced me into thinking about how to retest my application.
Despite all the suspicion that I was corrupting RAM with an array index overflow my detailed analysis of an obvious candidate procedure within my application found nothing. The issue was actually more obvious and did not need much looking for at all.
My problem is that I am using both ethernet and SD as shared SPI devices. Since I use ethernet for a web site, emails and daily UPD NTP time resets I made ethernet the default active SPI device. That means that SD card operations will fail or be problematic if you do not select the SD card for SPI when needed. It also means that if I don't remember to reselect ethernet for SPI when I am done with an SD card operation then there will be trouble there too. And if you are reading an SD card file line-by-line and writing it to a web page line-by-line (which I do) then your code has to continually switch the SPI devices.
So I rechecked all my SPI device selections and found the problem - everything worked OK.
So over this weekend I have reorganised my use of the SD card and reimplemented multi-level directory browsing with alphabetically sorted files (and file display) on my system's web pages. This also lead to some glitches when I did not get the SPI device (pin) selection correct - but it is all sorted now and all my system's log files are now being written into the required SD card subdirectories and I can browse the card and display any of the files via the system's web site.
Thanks again to those who helped get my mindset pointing in the right direction.
I think I just encountered this problem. Or at least I think my problem is similar to the one described here.
I use an Arduino Mega. All the SD functions fail and hang-up when my program reaches about 83K in size. After searching the Internet on this problem, I came across this thread on this forum. After browsing over the postings, I check my functions over and over again, I check my array indices again and did not find anything wrong.
Anyway, I just took Mr. fat16lib's advice and check the array indices again. I may have missed something, so to be safe I increased the all the arrays indices by 1 or more elements and all the SD functions start to works again. I think my problem is due the Array memory overflow. I have yet to figure out which array is causing the problem, but for now, I am just happy the problem is fixed.