Go Down

Topic: Reading Big CSV Files from SD Card becomes slower (Read 473 times) previous topic - next topic

Qar

I am reading one row of floats from a CSV file using "myFile.parseFloat()", doing few calculations and updating the color of 1600 LEDs. After that: printing the time needed. It needs to stay under 40ms for 25fps.

It's 1600 columns and over 20.000 rows. After around 8.000 rows, hickups occur.


Code: [Select]
for (int i = 0; i < NumberOfLEDs; i++)
  {
    int CurrentColor [3];
    CalculateColors(myFile.parseFloat(), CurrentColor);
    leds.setPixel(i, CurrentColor[0], CurrentColor[1], CurrentColor[2]);
  }
  Serial.println(ElapsedTimeInMS);
  while (ElapsedTimeInMS < FrameTime) {} //wait until 40ms passed
  leds.show();
  ElapsedTimeInMS -= FrameTime;
  Serial.println(LoopNr);
  LoopNr++;

Nothing to do with temps, because after restart it runs fine for 8k rows again.
Difficulty of calculations don't change, floats are the same size in the CSV file.

Do I need to flush the filestream sometimes?


If not, what's the reason for the slodowns?




Beside: Is there a better way than using a CSV file for storing floats on an SD-card?

Paul_KD7HB

Is there any change if you do not do the serial prints?

Paul

Qar

I would probably not notice anything, since it's just 20ms longer and there are not all LEDs in place yet. How should I check it and why should the serial print be a problem? Does it create some memory leak and the hickups occur, when RAM is freed? If this is the case, how can I free memory up manually?

Paul_KD7HB

I would probably not notice anything, since it's just 20ms longer and there are not all LEDs in place yet. How should I check it and why should the serial print be a problem? Does it create some memory leak and the hickups occur, when RAM is freed? If this is the case, how can I free memory up manually?
It all depends on how quickly you fill the serial buffer. When full, your program will wait.

Paul

Qar

Ah, okay, now i get what the flush() is for. So i could just flush everytime i filled the buffer. This way the serial print can be excluded as a reason, because the delay caused by the serial print must be always the same?

Qar

The problem seems to be related to "SD.h" library. I am using the beta "SdFs.h" library (https://github.com/greiman/SdFs) now: It reduzed the delay from around 30ms to 12ms and there are no hickups anymore.

Go Up