Scanning Analog Inputs and Writing to uSD Card

Forum 2005-2010 (read only) Hardware Interfacing
1

All,

I have a arduino nano (from Gravitech) and a uSD shield (from Sparkfun). I am trying to scan all 8 analog inputs as fast as possible and dump that data to the uSD card. I have a loop that gets the time, scans all 8 channels, and writes all of this to the card. The problem is if I do this loop 2000 times it works well; however, if I try to do this 3000 times it slows down a whole bunch and never writes anything to the file. I am using the SDFat library to perform the uSD stuff

Is this some internal memory (EEPROM) getting full??? If so, can I purge it??? My board has 30000 B of free memory. Which tells me that I can scan all 8 channels 3000 times to fill it up, coincidence??? (30000B=240000b=24000S=3000 loops of 8 channels)

Is it some limitation on file size???

I may try running different amounts of channel to see if it is based on the number of samples or channels or...

Thanks,
w102acd

Here is my sketch:
//THIS SKETCH IS USED TO BLINK AN LED (FOR STATUS),
//SCAN ANALOG INPUTS, AND WRITE THE DATA TO A
//MICROSD CARD AS FAST AS POSSIBLE.
//
//DATE : JUN 30 2010
//
#include <SdFat.h>
#include <SdFatUtil.h>
//
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
//
//**************************************************
void writeCRLF(SdFile &f)
{
f.write((uint8_t )"\r\n", 2);
}
//*************************************************
void writeNumber(SdFile &f, uint32_t n)
{
uint8_t buf[10];
uint8_t i = 0;
do {
i++;
buf[sizeof(buf) - i] = n%10 + '0';
n /= 10;
} while (n);
f.write(&buf[sizeof(buf) - i], i);
}
//**************************************************
void writeString(SdFile &f, char str)
{
uint8_t n;
for (n = 0; str[n]; n++);
f.write((uint8_t )str, n);
}
//************************************************
void setup(void)
{
Serial.begin(9600);
delay(2000);
//
pinMode(2, OUTPUT);
digitalWrite(2, HIGH);
//
int j = 0;
for (j = 0; j < 21; j++) {
delay(250);
digitalWrite(2, LOW);
delay(250);
digitalWrite(2, HIGH);
}
//
Serial.print(!card.init());
Serial.print(!volume.init(card));
Serial.print(!root.openRoot(volume));
//
char name[] = "DATA00.CSV";
for (uint8_t i = 0; i < 100; i++) {
name[4] = i/10 + '0';
name[5] = i%10 + '0';
if (file.open(root, name, O_CREAT | O_EXCL | O_WRITE)) break;
}
Serial.println(!file.isOpen());
//
delay(2000);
digitalWrite(2, LOW);
delay(2000);
digitalWrite(2, HIGH);
//
writeString(file, "Microseconds,AI0,AI1,AI2,AI3,AI4,AI5,AI6,AI7");
writeCRLF(file);
//
for (j = 0; j < 3001; j++) {
writeNumber(file, micros());
writeString(file, ",");
writeNumber(file, analogRead(0));
writeString(file, ",");
writeNumber(file, analogRead(1));
writeString(file, ",");
writeNumber(file, analogRead(2));
writeString(file, ",");
writeNumber(file, analogRead(3));
writeString(file, ",");
writeNumber(file, analogRead(4));
writeString(file, ",");
writeNumber(file, analogRead(5));
writeString(file, ",");
writeNumber(file, analogRead(6));
writeString(file, ",");
writeNumber(file, analogRead(7));
writeString(file, ",");
writeCRLF(file);
//
Serial.println(j);
}
file.close();
digitalWrite(2, LOW);
}
//**************************************************
void loop(void) {}
//**************************************************

2

It is writting an integer (bits?) and 8 - 10 bit channels.

The uSD card is 2GB and formatted FAT32 using Windows XP.

Also, I found this thread whuch seems similar (and unresolved).
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1261141357

Basically, other people were having a similar problem and speculate that it might be a character limitation in the SDFat library.

Thanks,
w102acd

3

My board has 30000 B of free memory

You board probably has 30 000 bytes of free program memory, which is completely irrelevant here - you need RAM for this, not ROM.

4

New developments:
If I run the program enough, it causes issues with the writing data to the file.

If I run my sketch once at 2500 loops it will work fine; however, if I run it a second time it will not write it correctly. It will gererate an empty second file.

I know I am not running to the end of the capacity because it is a 2GB card and I can successfully write a 100KB file, but not a second one. Could it be some kind of partition size or something like that???

Right now I am using the Windows formatting utility. Should I be using something else???

As far as the amount of data I am writing. Do you mean the amount of data that I am writing per command or the amount that the entire program will write??? Eventually I want to have the capability to run until the card is full (several hours).

I get good results when it works. I have AI4 shorted to ground and AI5 to the 3.3V signal and they both look nice and smooth.

Thanks!!!

5

I think part of your problem involves the creation of the file name and open function itself.
First, the number should be converted to an ASCII equivalent character. Is the system function atoi() available ?
Does the second run actually create a file? If so, what is the name?

6

The creation of the file name and open function are straight from an example included in the SDFat library.

I have never heard of a atoi() function, but will search for it and see.

The file system has worked in the past. If I drop the number of loops to 500 I can create multiple successful files before it breaks. If I run 2500 loops it breaks somewhere in the second file.

I read in another post that writing a file this way is not good. I remember something about the way an SD card works is that it will rewrite the entire file everytime you write something to that file. I remember something like "and eventually it will stop working." It works well for relatively small amounts of data.

Should I be using the raw write instead??? I was looking into it. It appears to have the capability to write much faster; however, I assume that the output file will be in binary. Is converting something from this format to an ascii format difficult??? I would store all of the data on the card and then do any "post-processing" on a computer.

I want to use this as a rocket flight computer/data recorder so the faster I can scan the channels the better off I will be. Our other more complex system scans at 500 Hz and I was hoping to get that here and be able to write this to the card as long as the card has space left.

Thanks,
Andy

7

Should I be using the append function instead of the write function???

Andy

8

All,

Well, I resolved the apparent write limit. If I format the card using the recommended format program it will write and write and write.

I have successfully written a loop of 60000 (unsigned int) without any issues. I will test overnight using more loops and an unsigned long or nested loops. I estimate that my card will hold ~70 hours worth of data.

Now on to the next problem:
I would like to scan as fast as possible at a fairly consistent rate. I ran two times with 60000 loops and there is a whole bunch of bouncing around in the amount of time it takes for each loop. The average is around 0.0042 seconds with a 0.0038 minimum and 0.3918 MAXIMUM. I have some excel plots of this, but have no idea how to post them.

Anyway, I would like to understand why this is all over the place. There are big spikes (0.4-0.2 seconds) every 50 seconds and small spikes (0.005-0.01 seconds) every 0.05 seconds. This tells me that something slows it down and really slows it down consistently after so much data, sampling, etc????

Thanks for all of the help so far!!!

9

I want to use this as a rocket flight computer/data recorder

I'm curious to know what sort of amateur rocket needs recording over more than 50 seconds.

10

If I format the card using the recommended format program it will write and write and write.

What is the recommended formating program, please?

(And I loved AWOL's question! But imagine he wasn't serious. If he was, I agree with what I infer as w102acd's view: that even if it is only going to run for 50 seconds, "loose threads" (like unexplained crashes) need to be resolved.)

11

Oh, I noticed that you should probably read ALL the inputs first before writing the data -- that would make the data written at the stated time more correlated. And consider using flush() after the write -- the system could be buffering the data before actually physically writing.
That would account for large differences in processing time.

12

This is how I anticipate a rocket launch going:

  1. Turn on Arduino and start collecting data
  2. Arm the rocket and leave the launch area (5-10 min)
  3. Launch the rocket and track using GPS telemetry (5-10 min)
  4. Use GPS telemetry to track down the rocket (???)
  5. Drive to Billy Bob's house after he finds the rocket and takes it to his house and hasn't turned off the Arduino yet (ha, ha)

Reasonably big rockets will drift a whole bunch under parachute. If you are really unlucky, after the rocket hits the ground the wind will drag the rocket all over the place.

Another nice feature would be that I could do a whole bunch of launches in one day and never have to download the data. Probably a bad idea!!!

In reality, I have a 2GB card and I want to be able to use it all, otherwise I would have bought a smaller card.

The formatter is from Panasonic and can be found on www.sdcard.org or something like that. If you have the SDFat library downloaded it is in the readme file in there. If you still can't find it, let me know and I will send a link. I don't have anything on this laptop.

I will try the flush command. I was going to look into the libraries and see how big the buffers are. This is exactly what I was thinking. I tried adding a 2 millisecond delay at the end of each write and the problem just takes longer to develop.

The raw data solution does not seem easy. Is there anything that can be done to speed up the "disk driver"???

The next question:
What is the fastest way to scan and write this data???

  1. Leave my loop the same
  2. Scan all channels and then write it all out (better timestamp, thanks ideeasman)
  3. Could you scan all of the channels and then load a string variable with the data and only write a single variable rather than 9 times???

Thanks,
Andy

13

Alright, I looked at the flush() command. Can it be used with the SDFat library??? The only thing I could find was a Serial.flush(), but I am not using Serial at all in my sketch.

Would trying to increase the buffer size help at all or just push the problem to later and less frequent???

Thanks,
Andy

14

Another interesting question about the raw data stuff. If I had another uSD card installed, could I write the raw to uSD1 very fast and consistently. And then when I tell it to, read it back in and spit it out to uSD2 in ASCII format. Then I would have my cake and eat it to. Seems tricky, but doable???

Could this even be done on the same card??? I don't really understand the whole raw data thing.

Thanks,
Andy

15

Do you mean

  1. run 2 sd cards while in flight, or
  2. log in raw format while in flight, then, at the end of the day, transfer all that data over to another card in a more friendly format?

If you mean the first, there wouldn't be much point, trying to get two cards working will be difficult with just one hardware spi bus for a start, the fact that they'll be fighting for time on bus would mean you'd be getting a lot less samples too.

Number 2 could be the way to go. Basically, when you're writing in RAW format your just treating the sdcard like a very large eeprom, writing the bytes direct to the card. You're not worrying about files at all so there's little overhead, which if you're wanting to get as many samples as possible would probably be better.

What you could do is:
Have your flight data being written in RAW format at will to the card during your days rocketeering.
Then have another sketch for when you get home that spits the data out from the sdcard (while pluged into your arduino) over the serial to a processing sketch which then formats the raw data in a nicely formatted way that excel can read. If you really need as many samples per second as possible, then I'd suggest that this is the way to go.

Just to throw another curve ball in there, have you tried the filelogger library? Google Code Archive - Long-term storage for Google Code Project Hosting.
It's a very light weight way of writing FAT files to sdcards, a lot less overhead than the SDFat library I believe.

I've used it myself and had no problems.

The Cageybee

16

Thanks Cageybee!!! I always forget to take advantage of the Serial stuff. If I read your reply correctly, it would be the best to write in the raw data format to the SD card to be able to scan fast and consistently. Once I have the data on the card I could load a second sketch (or have another arduino ready) and print all that data out to the serial port while capturing it with VB or something (or I could write it to another card). Anyone have a link to a code that will read the serial stuff and put it in a file for VB6. BTW I know that VB6 is an old code, but I still use it a ton.

I am going to play around with some other things today, but may give this a shot over the weekend.

Thanks,
Andy

17

If you're going to be processing the data using VB6 you could just use hyperterminal, or any other terminal software to just capture the serial data to a file, then use VB6 to parse and split the data up in to meaningful data.

There are a few things you need to think about first though, namely how you want the data to be stored and whether you want a timestamp for each pass of samples.

Basically, what's your data structure going to look like.

So, say you wanted to store the 8 analog inputs values as 'int's and have a timestamp generated by the millis() function, which is an 'unsigned long', each loop will be: (8 * 2bytes) + 4bytes. So that's 20 bytes total.

Now, sdcards are split into 512byte blocks, so you'd have to use a buffer to store the data, then once the data's full, write the data to the card, increment a counter to write the next block to, then start overwriting the buffer, till it's full, and so on, and so on.

To simplify things a little you'd probably be better off writing the data to the card when you've got 500bytes of data and just leaving the final 12bytes as 'junk'. So that means you'll be storing the data from 25 'loops' to the card each time.

To be honest, I've never written to sd in RAW before, but if you give me a better idea of what you want your datastructure to look like, I'll look into it and come up with some code you can use/adapt.

Regards,
The Cageybee

18

Okay. So I've had a little look see into this and think I've come up with something.

There are a couple of RAW SD libraries available, but I couldn't get one of them to compile. A few people are having the same problem with it, but no solutions of how to resolve it yet.

I did get the one from here Download sdcardlibrary.zip (Arduino weather station) to compile though.

Assuming you wanted the data structure I outlined above, you could use the following code to write the data to the card.

#include <SDCARD.h>

volatile unsigned char buffer[512]; // Read/Write buffer
unsigned char bufferPos = 0; // Current position in the buffer
unsigned long sector = 0; // Current sector
int error = 0; // Error flag

void setup(){
Serial.begin(9600);
}

void loop(){
sampleInputs();
}

void sampleInputs(){
int sampleData; // Raw sampled data
byte loByte, hiByte; // Hold raw data split into hi & lo bytes
for (int i = 0; i < 8; i++){ // Sample all inputs, spilt into hi & lo bytes, push to buffer
sampleData = analogRead(i);
loByte = sampleData &0xFF;
hiByte = sampleData >>8;
buffer[bufferPos] = loByte;
bufferPos++;
buffer[bufferPos] = hiByte;
bufferPos++;
}
unsigned long timeStamp = millis();
buffer[bufferPos] = timeStamp &0xFF; // Split timestamp up into 4 bytes and push to buffer
bufferPos++;
buffer[bufferPos] = timeStamp >>8;
bufferPos++;
buffer[bufferPos] = timeStamp >>16;
bufferPos++;
buffer[bufferPos] = timeStamp >>24;
bufferPos++;

if (bufferPos >=499) {writeToSD();} // If buffer full, write it to the SDcard
}

void writeToSD(){ // Writes data to card, increments sector, resets buffer
error = SDCARD.writeblock(sector); // Write data in the buffer to the card at current sector
if (error !=0){ // If an error occured
Serial.print("sd card write error... code = ");
Serial.println(error);
}
else { // If write was successful
sector++; // Increment current sector num
bufferPos = 0; // Reset buffer position ready to overwrite
}
}

Then when it comes to reading the data back, you can reassemble the data, put in in a nice format and spit it out over serial. Say you wanted to save it as a CSV file, you can just format the data in that format as you send it over serial and just capture the data to a file using hyperterminal or similar.

I've got a couple of ideas of things you could do.

If you've got a free digital pin you could have a sliding switch that selects whether you want to write data to the card or dump the data over serial.

Also, if you were to store the sector position to EEPROM, between launches you could turn the logger off, then when you're ready to launch, turn the logger on, the current sector is loaded from EEPROM and the data continues to be written from the last sector so you don't overwrite your data. You could have the logger write 1 sector of 0xFF's to the card on startup so you can tell when a new logging session has started.

Are you using a GPS? Or are you just making analog samples?

Let me know if you want me to go any further with this.

Regards,
The Cageybee

19

Cageybee,

You are getting way ahead of me. I've got some catching up to do. I'll do my best to keep up.

A little more testing of the old way revealed that is it not a buffer issue, I think. I put a delay of 20ms inside the loop. The spikes were further apart but occured after the same amount of data was written to the file as with no delay. If it was a buffering thing this would get better the slower and slower I tried to write things to the file, right???

I am using the SDFat library and it has a raw write example included with it. I will try to get it working and see what happens. I will also take the time look through your code and make it makes sense to me.

Crowley,

I was also thinking about ways to trigger things to start recording. The problem is once something is detected, I want to start recording slightly earlier. In rockets things happen extremely fast. The GPS modules that I have seen all run at very low frequencies (~10 Hz). Before the GPS detects that the rocket moved, I could be half way to motor burn-out.

My father-in-law (a pilot) has a saying that I can use here. There is no such thing as too much runway. In this case, there is no such thing as too much data. I would rather have 70 hours of data with 10 useful seconds than 8 seconds of data with 10 seconds of useful data. Data is easier to delete than generate.

Thanks again for everyone's input and help,
Andy

20

At what amount of data does the spike occur? BTW, it's the amount that causes the physical write to take place, rarely time.
The recent suggestions about storing data and writing out 500 byte blocks IS buffering -- internally by your program.
I'm guessing ,but are the spikes at .5k, 4k or 8k ?