Hi to everybody! I've been following this forum for a while. As my first post, i wanted to show you an issue i've been dealing lately.
My final goal is to realize a fast data logger , collecting data from an MPU 6050 and a GPS, and store them into an SD.
It's obvious that the SD writing time is going to set a lower bound on how fast i could go with the sampling time of the sensors data.
The main issue with it is that, once in a while, writing and synching time of the SD randomly goes as up as ~100-200ms.
I'm attaching here the routine i've been using for debugging this issue:
NB queue is a buffer long 64bytes, and BUFFER_SIZE is defined as 64, while FILE_SIZE is the number of writes i allow, for debugging purposes.
FILENAME_TEST is defined as "TEST.BIN"
void log_data()
{
// Open File
data_row.open(FILENAME_TEST, O_CREAT | O_WRITE);
// Logging loop has begun!
delay(1000);
Serial.println(F("Starting logging routine"));
Serial.flush();
delay(10);
uint32_t bn = 0;
uint32_t dt_write;
uint32_t dt_sync;
uint32_t max_dt_write = 0;
uint32_t max_dt_sync = 0;
struct timer {
uint16_t st;
uint16_t old_st;
uint16_t max_st;
} sampling;
uint16_t bytes_written;
uint32_t max_bytes;
bool sync_flag = false;
// Fill buffer
for(uint8_t i = 0; i<BUFFER_SIZE; i++)
{
queue = i;
-
}*
-
// Free cache*
uint8_t cache = (uint8_t)SD.vol()->cacheClear();
-
delay(1000);*
-
if(cache == 0)*
-
{*
-
Serial.println(F("Cache clear error"));*
-
}*
-
while(sd_ready)*
-
{*
-
sampling.st = millis() - sampling.old_st;*
-
queue[0] = bn;*
-
sampling.old_st = millis();*
-
if(sampling.st > sampling.max_st)*
-
{*
-
sampling.max_st = sampling.st;*
-
}*
-
digitalWrite(WRITE_PIN, HIGH);*
-
dt_write = millis();*
-
// Writing full array*
bytes_written = data_row.write((uint8_t*) queue, BUFFER_SIZE);
-
digitalWrite(WRITE_PIN, LOW);*
-
dt_write = millis() - dt_write;*
-
dt_sync = millis();*
-
bn++; // Update written data block*
-
// FILE SIZE ROUTINE*
-
if(bn == FILE_SIZE)*
-
{*
-
break;*
-
}*
-
if(sync_flag)*
-
{*
-
digitalWrite(SYNC_LED, HIGH);*
-
bn = 0;*
-
data_row.sync();*
-
digitalWrite(SYNC_LED, LOW);*
-
}*
-
dt_sync = millis() - dt_sync;*
-
//Update current max*
-
if(dt_write > max_dt_write)*
-
{*
-
max_dt_write = dt_write;*
max_bytes = bn * BUFFER_SIZE;
-
}*
-
if(dt_sync > max_dt_sync)*
-
{*
-
max_dt_sync = dt_sync;*
-
}*
-
// When user inputs to the serial, then stop logging*
-
if(Serial.available())*
-
{*
-
Serial.println(F("Stop LOG"));*
-
break;*
-
}*
-
}*
-
// Close file*
-
uint32_t dt_close;*
-
Serial.println(F("Closing file..."));*
-
dt_close = millis();*
-
data_row.close();*
-
dt_close = millis() - dt_close;*
-
Serial.print(F("Closing file took -> ")); Serial.println(dt_close);*
-
Serial.print(F("Max writing time -> ")); Serial.println(max_dt_write);*
-
Serial.print(F("Max sync time -> ")); Serial.println(max_dt_sync);*
-
Serial.print(F("Sampling Time -> ")); Serial.println(sampling.max_st);*
-
Serial.print(F("Written bytes -> ")); Serial.println(bytes_written);*
-
Serial.print(F("Bytes written when MAX -> ")); Serial.println(max_bytes);*
}[/quote]
Through an oscilloscope i've found that, after i've wrote 512bytes worth of data, sync method is automatically called. I think the problem is related to the actual cluster size of my sd card, which is 32kb (formatted using SDFormatter, as the top topic on this forum suggested).
Any suggestion about this is appreciated.
Can you post detailed received results?
As far I know, SPI bus clock is limited to 20-25MHz for most of SD/MMC card, then you can't get any better performance whatever card class is rated. You are further limited to the master (ATMega) clock and preparation delays before sending command and after receiving result that may drop moderately.
The fastest writing speed with SPI bus is when work on maximum allowed SPI speed and SD allow receiving multiple blocks. I'm not certain exactly what capabilities and performances are with the lib, but it is based on third party FAT lib.
Thank you for your answer.
As a matter of fact, i'm using straight the SdFat library, since it looks like the built-in SD library with the Arduino IDE is no more than a simplified wrapper for it.
Right now i'm dealing with this issue by buffering the data from the sensors and write a chunck of 128bytes every 200ms , reaching a sampling time of 10ms. The writing adds an overhead of 5ms every time a write calls is made, but it is good enough.
Also, i initialize the SD using the SPI_FULL_SPEED command, that is going @8MHz (watching the SCL line with an oscilloscope, it shows a time between rising edges of ~120ns).
Following your thought, i'm handling the SD card at a speed that is almost 1/3 of its capability. Am i right ?
I have made my plain C lib for 8-bit PIC long time ago, perhaps 6 or 7 years. At that time I read almost all related to SD card (including SD card association documents). That is long time ago and I believe many things are changed according to SD card technology evolution. I can only suggest what I remember clearly from that time.
When initialize card, it is recommended to start with SPI speed between 100 and 400KHz, i.e. low. That is guarantied to work with all card classes. When successfully initialized, you can collect card info (type, size, max speed etc) and then you may rise SPI speed you have collected from card info, or if not possible because limitation of MCU performance, you may try to lower it or rise step by step from 1Mbit to maximum SD card SPI speed and try read/write.
As long I can remember, the smaller block size is 32b and max. 512, but all these values may be collected from card to or recent SD card documentation from www.sdcard.org to be sure exactly.
After a couple of attempts, i discovered that actual cluster sizes of modern SD are 16kb or a multiple of this.
Using a 32GB SD, a delay of 200ms between write calls(regardless of writing 64 or 128 bytes per time), ensure that it takes no more than ~5ms.
Using an older 512MB SD, 100ms are more than enough.
I'm going to post the full code integrating the MPU and the GPS asap for your reference!
I'm glad you have make it to work!
News!
As i stated a few post above, the SD latency problem needs to be dealt using a FIFO buffer storing the incoming data in an ISR, while the SD is busy being written.
I've implemented this using NilRTOS since it allows to implement the above mentioned idea in a faster and cleaner way.
I'm attaching here the source code using so far. I've tried to implement the reading from the GPS too (connected via SoftwareSerial. I have a custom implementation of it , allowing to manipulate its internal buffer so to avoid wasting additional 64/128bytes for another buffer).
GitHub Rep
Anyone has a thought on how to proceed or if it is feasible on an Arduino Nano ?
Any help is appreciated.