Processing Large SD card Files

Projects Programming
1

Hi All

I am using an arduino Mega with an external SD card module. I have a large file on the SD card (7 840 KB, or 4 values logged at 10Hz for about 8 hours) that needs to be processed using a predetermined algorithm.

The algorithm works for almost 5 of the 8 hours but then exits the algorithm. The processing function is run in the main loop when the user enters a value in to the serial monitor. The start of the processing function is as follows:

void processNight(String FileName){
  processFile  = SD.open(FileName);
  if(processFile){
    startTime = millis();
    while (processNightFile.available()){
      currentTime = processNightFile.parseFloat();
      currentSPO2 = processNightFile.parseFloat();
      currentFlow = processNightFile.parseFloat();
      currentBE = processNightFile.parseFloat();
      currentHR = processNightFile.parseFloat();
     //The processing is done here
    }
   //Some print function that shows how long the function ran for. 
}
}

The function runs for almost 1028 seconds and then exits. Is there any way to extend this cutoff time, or do I need to break the file up and process it in chunks?

2

your are opening processFile but using processNightFile ?

Don't post snippets (Snippets R Us!)

--
you would probably be better off to record the data in binary form, you will save tons of time by not using parseFloat

1 Like
3

What type file system on the SD card? 7840kb is too large for fat32.
Sorry was thinking gb,not mb.

4

Sorry, I can't share the whole function because it is quite long and not the point of the question.

void processDiagnosticNight(String FileName){
  processNightFile  = SD.open(processNightFileName);
  if(processNightFile){
    startTime = millis();
    while (processNightFile.available()){
      currentTime = processNightFile.parseFloat();
      currentSPO2 = processNightFile.parseFloat();
      currentFlow = processNightFile.parseFloat();
      currentBE = processNightFile.parseFloat();
      currentHR = processNightFile.parseFloat();

    //Processing algorithm 
   }
  }
 }

I can't record the data in binary form because of external constraints. What is causing the function to exit prematurely and how can I avoid this?

And yes, sd card is using FAT32 file system

5

IF the problem was mine, I would count the number of loops in your "while" loop and display that count to see if the while stopped at the same count every time.

6

could you attach the file here or does it include confidential information?

you could record both in ASCII and binary and use the binary version for the processing

this code

checks if 1 byte is available and reads lots of bytes without any error detection. if for whatever reason the recording was incorrect, you have no way to recover.

if you don't want to go binary, I would make sure that the records are separated by an identifiable marker and terminate the record with a new line. I would read the line in (read until '\n') and then parse the line performing some error detection.

7

Why are you doing such processing on an Arduino?

Wouldn't a PC or similar be more appropriate?

8

What does file.available() return? The docs say it's not a bool, but an integer. So is it 32767 until the the remaining file is less than that, or is it the just the least significant 15 bits of the available size (which could be zero when there's more file actually available?

9

Is the data formatted as lines of text? Perhaps read into a buffer, one line at a time, until read() returns -1 indicating end of data.

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.