SD logging time blocks (Motorcycle data logger)

Using Arduino Storage
1

Hello,

I am looking for specialists for SD card logging with the Arduino device. I am currently working on a motorcycle data logger project which is almost managed. I use an arduino Mega with an adafruit microSD breakout board to log data.

Almost managed because I have a problem when I try to log the data on the SD. It consumes sometimes to much time (1s) to write data but I am not able to understand why....

Take a look to the post #127-#130 it is a good summary of the problem.

In my sketch there are 2 mains loops one at 100 Hz for fast sensors and the other at 20 Hz for the GPS. Hence available time to write data is sometimes to short between two readings that's why I use a yield function into the Sdfat library.

The weird is that it works perfectly but after around 70 seconds write data take a very very long time. Hence lot of sensor readings are missed :confused:

I hope I will find help there.

Thank you in advance,
Pm

2

powergravity:
Hello,

I am looking for specialists for SD card logging with the Arduino device. I am currently working on a motorcycle data logger project which is almost managed. I use an arduino Mega with an adafruit microSD breakout board to log data.

Almost managed because I have a problem when I try to log the data on the SD. It consumes sometimes to much time (1s) to write data but I am not able to understand why....

Take a look to the post #127-#130 it is a good summary of the problem.

In my sketch there are 2 mains loops one at 100 Hz for fast sensors and the other at 20 Hz for the GPS. Hence available time to write data is sometimes to short between two readings that's why I use a yield function into the Sdfat library.

The weird is that it works perfectly but after around 70 seconds write data take a very very long time. Hence lot of sensor readings are missed :confused:

I hope I will find help there.

Thank you in advance,
Pm

SdCards are MCU also. The Arduino libraries use the simplest access method possible. The FLASH memory inside them requires page erasures before their content can be changed. This takes time.

When you extend a file, the FAT structures have to be adjusted(erased/rewritten). Depending on how the SDCard was formated (Cluster/sector sizing), adding a single byte to a file could cause thousands of bytes to be written. If you can pre-allocated your sample, and audio files all of this overhead can be done once.

//SdFat.h

bool FatFile::createContiguous ( FatFile *   dirFile,
 const char *   path,
 uint32_t   size 
 );

You are going to have to code your application to handle these pauses. I have done it by sampling during an interrupt, storing the data in a memory buffer. The foreground loop() looks for dirty buffers and writes them to the SDCard as fast as it can. As long as the sampling buffers are cleared faster than they are filled everything works ok.

You need to calculate your dataflow rates.

Chuck.

3

Thank you very much for yours answer. It's very motivating to find help.

//SdFat.h

bool FatFile::createContiguous ( FatFile *  dirFile,
const char *  path,
uint32_t  size
);

I didn't understand how I could integrate it into my project. It is not so easy when you are not so familiar with SD properties.

You seems to be a "storage" specialist. Hence could you give me your feedback about the fastest micro SD usable with SDfat library?

Thank you,
Pm

4

powergravity:
Thank you very much for yours answer. It's very motivating to find help.
I didn't understand how I could integrate it into my project. It is not so easy when you are not so familiar with SD properties.

You seems to be a "storage" specialist. Hence could you give me your feedback about the fastest micro SD usable with SDfat library?

Thank you,
Pm

Every SDCard will pause while they actually store the data into the Flash Array. You will have to account for these pauses.
High speed SDcards that are used to store 4k video have special modes of data transfer,to support megabytes per second. The Arduino are memory limited, they do not have the resources to support these high speed modes. The backwards compatible (SPI) modes the Arduino's use are slow.

Most Class 10 cards are faster than the Arduino.

You are going to have to write code that accommodates the Arduino's limitations. Or use a faster MCU.

Raspberry Pi's are faster, more robust. But, they have their limitations, slow Boot, respond badly to power fails, not deterministic.

Chuck.

5

I am not able to understand how to pre-extend a log file.
Some help would be appreciated.

Raspberry Pi's are faster, more robust. But, they have their limitations, slow Boot, respond badly to power fails, not deterministic.

I am very interesting to send data to the Pi. I read that it's very complex to set the Pi as a SPI or I2C slave whose the task would be to write data on the SD as soon as they would be available on the bus.
However I saw that interface arduino and Pi with serial communication is very easy but I wonder if it's enough fast in my case.

Thank you,
Pm

6

Hi,

After some investigations I saw SD initialization for fast data logging looks like:

 // initialize the SD card at SPI_FULL_SPEED for best performance.
  // try SPI_HALF_SPEED if bus errors occur.
  if (!sd.begin(chipSelect, SPI_FULL_SPEED)) {
    sd.initErrorHalt();
  }

  // delete possible existing file
  sd.remove("RawWrite.txt");

  // create a contiguous file
  if (!file.createContiguous(sd.vwd(), "RawWrite.txt", 512UL*BLOCK_COUNT)) {
    error("createContiguous failed");
  }
  // get the location of the file's blocks
  if (!file.contiguousRange(&bgnBlock, &endBlock)) {
    error("contiguousRange failed");
  }

According to my SD storage experience I have some difficulties to integrate it. :confused:

Thanks to a moderator who makes my sketches usable without all the sensors, hence you can find it attached with the associated library. You just need a SD board.
I am actually using a 16 Gb SanDisk Extreme Plus whose the bench.ino test gives:

Type any character to start
FreeStack: 6699
Type is FAT32
Card size: 15.93 GB (GB = 1E9 bytes)

Manufacturer ID: 0X3
OEM ID: SD
Product: SP16G
Version: 8.0
Serial number: 0XD46709D8
Manufacturing date: 8/2015

File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
262.80,120024,1484,1941
259.44,192484,1488,1966

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
484.56,2040,996,1050
483.90,2040,1000,1051

The attached sketch perfectly illustrates the timing problem when the recording file is extended leading inevitably to missing writings.

With the real sketch I get:

1281×775 17.3 KB

With the "sensor simulated" sketch which is attached I obtain:

Press 1 to start logging, 0 to stop
Initializing SD card...
  SD card initialized.
Writing to data_03.txt
W5330396=5
S5330828=5
I5331352=6
W18860092=2
S18860568=2
I18867564=2
W18946268=2
S18946744=2
I18947312=2
I18974124=3
W26022492=2
S26022976=2
I26033992=4
W47646692=2
S47647176=2
I47652068=3
I47670404=2
W69255272=2
S69255752=2
I69259712=3
I75820696=2
W101575040=2
S101575568=2
I101577560=3
W101595128=2
S101595660=2
I101599992=3
I102401784=2
I102488544=2
W102568180=2
S102568704=2
I102569336=2
W102641836=2
S102642356=2
I102648856=3
.....

Each line is a missing writing. The letter corresponds to the sensor's name (I=IMU, S=Steering sensor, W=Wheel speed sensor) and then the time in micros when it happens.

I would be grateful if someone could help me. It's the last part of a 2-month project.

Thank you very much,
Pm

simple.zip (157 KB)

7

what about an intermediate fram array... i have a prototype comming in that i designed that is 16Mbit fram memory i am looking at connecting two arduinos to it at the same time on a round robbin sort of system basically one arduinio (probably a wifi enabled uno with sd taps into the fram array on a read only basis it reads all the data on the array and then copies it to sd that is its whole function, the other arduino (mega ) collect information from 128 12 bit ADC, pushes information to 16 12 bit DACs, controls 996 pwm outputs and 512 digital outputs. the fram sharing thing is new to me but i have seen examples on google where this works, i just havent worked out all of the control protocols in the "me" way. the prototypes for the fram modules for 16Mbit is gonna cost me something like 20$ a piece populated but 20$ is worth the lessened head ache... js.

8

Hi,

worth the lessened head ache

For sure :stuck_out_tongue:

But I am looking for a quick solution. So as chucktodd has suggested pre-allocated the sample could be the solution of my problem.
But I have lot of difficulties to integrate it in my data logger sketch. :confused:

Any suggestion to proceed?

Have a good day,
Pm

9

I managed to pre-allocate file size but it leads to other problems. I get anarchic characters at the end of the recording txt file :-\

Please take a look to my last post.
I am lost...

Maybe you could help me.

Thank you,
Pm

10

powergravity:
I managed to pre-allocate file size but it leads to other problems. I get anarchic characters at the end of the recording txt file :-\

Please take a look to my last post.
I am lost...

Maybe you could help me.

Thank you,
Pm

Preallocate the file.
Write your data to the file.
Truncate the file when you have written your last byte.

Chuck.

11

Hi,

Thank you for your advices.

According to them: I changed the pre-allocated size to 5MB and then 10MB, I wrote the data in the file and I finally truncated the recordings. I paid attention to format the SD with the utility included into SDfat library before the test.

The results are worse :relaxed:
Take a look to the logging time.

with a pre-allocated size of 5Mb:

1225×742 15 KB

with a pre-allocated size of 10Mb:

1283×792 15.1 KB

with the default pre allocated size (initial case):

1216×754 14.7 KB

This logging problem is a real headache!

Maybe another suggestion about the cause of this long logging time?

Thank you very much,
Pm

12

what about a faster arduino?

13

Amazing I bought a Due ten minutes ago :slight_smile:

Have a good day,
Pm