Scanning Analog Inputs and Writing to uSD Card

I am assuming 20 bytes are written each loop. 8*2 for the integers plus 4 for the unsigned long. There are really two different delays in writting the data. The first one occurs every 12 or 13 loops and are ~0.05 seconds apart. This one has a time step of ~0.007 seconds and is not too big of a deal. 12 or 13 loops give me 240 and 260 bytes between the small delays.

The large delays occur every ~13100 loops and this delay is 0.2-0.4 seconds long. This one is my major concern. 13100 loops gives 262 kB between long delays and 56-57 seconds.

It appears to me that I have a factor of two missing somewhere. If I multiply the number of bytes by a factor of two I get: 1. 480-520 bytes between small jumps 2. 524 kilobytes between big jumps These are both dangerously close to 512 bytes and 512 kB.

My questions are: 1. Is there a factor of two that I am missing somewhere??? 2. Is there a fix for this problem??? a. Writing a raw data file??? b. Writing in blocks of 512 bytes instead of small increments??? 3. Why did this problem not go away when I put a 20ms delay in each loop???

Thanks, Andy

I found the factor of two. The code does not write things as integers, it writes everything as uint8_t. The time is 8 or 9 bytes, each analog is 1 to 4 bytes, and the strings are 11 bytes. The total for my case is either 41 or 42 bytes per loop.

I still have the other questions.

Thanks, Andt

If the rocket goes out of range of the telemetry or telemetry is lost, I will not have the GPS data. It also can be used as a crude tracking device. If for some reason the rocket gets absolutely lost, you could just drive around until you get close enough to the rocket to acquire a signal again.

This may be some very good news. I looked into the rawrite example with the SDFat libraries. It compiled and ran smoothly. If what the sketch tells you is true this would be impressive. It says that it it writing 51200 bytes of data per second for 100 seconds.

So if this alone is true, it is something great.

Then just for curiosity sake, I put the card in my computer and it has an ascii readable file. It is 5000KB in size and took 100 seconds which is 50000 bytes per second. If I can get this data rate out, I will have over 1200 hz or 0.83ms time step. Right now I cannot successfully get 4ms to work.

I will look into it more when I get time, but this looks very promising!!!

Does that include the reading of the 8 analog devices, too? I wrote a sample read that uses pointers. It takes 900 usec to read and store the 8 values in the buffer (NOT WRITE to SD), using 20 bytes binary.

I haven't tried to time the sampleInputs() listed in this thread. (I think mine would be faster -- It has no loops for creating ASCII data.)

That is fast!!! I think writing to the SD card appears to be a bigger bottleneck than scanning. In my old code, if I had it just write out meaningless numbers instead of scanning it was around 700 us faster per loop.

This will allow me to do other stuff on top of scanning the analog channels while still getting a high enough frequency.

Well, I have made some progress towards this. I have a sketch that writes time (microseconds) and all 8 analog channels. I can get about 200 Hz out of it semi-consistently.

Here are the few issues I am having now:

  1. I can get 200 Hz out of it with the exception of one point at the very front of the file that drops to 50-150 Hz. It is always one point and always the 31st line. I am doing 10 lines per 512 byte block so this would be the first line of the third block. I suppose it could be the writing of the second block that really causes this. Why does it do it there and nowhere else???

  2. I can only seem to get 75520 good lines out of this sketch. I'm not really sure why. Line 75521 contains a whole bunch of "NUL" as viewed in Notepad++. Any suggestions on getting over this hump??? The files I am generating are 5000KB in size and the card is 2GB so I know I am not running out of space. After 3866624 everything goes to NUL???

I also noticed, I can only get 250-200 Hz maximum out of it. 200 for line 0-8 and 250 for line 9. It still has the one point dip in the front.

Thanks, Andy

Just confirmed the number of "NUL" characters is equal to the remaining characters that should have been written to the file.

Any chance you could post the current sketch and some of the file that's stored on the sdcard.

A few lines of 'good' recordings and the bad ones.

Would be even more useful if you could post them as read by a hex editor, as then it's easier to see exactly what bytes have been written to the card.

The Cageybee

Here is a truncated file (not really code):

22679564,0480,0417,0407,0317,0000,0830,0576,0513
22684576,0558,0547,0550,0497,0000,0831,0761,0687
22689580,0724,0722,0720,0675,0000,0830,0834,0779
22694592,0622,0568,0539,0511,0000,0830,0768,0739
22699596,0681,0631,0593,0557,0000,0829,0801,0774
22704596,0890,0895,0881,0836,0000,0830,0883,0858
22709600,0814,0800,0786,0760,0000,0830,0813,0799
22714612,0718,0688,0671,0660,0000,0829,0791,0783
22719620,0880,0893,0892,0873,0000,0830,0874,0862
22724628,0881,0893,0898,0889,0000,0830,0850,0850            
22729636,0702,0667,0653,0659,0000,0829,0767,0775
22734644,0798,0788,0778,0770,0000,0830,0836,0835
22739652,0902,0914,0911,0894,0000,0830,0858,0854
22744660,0738,0710,0694,0692,0000,0829,0772,0773
22749668,0744,0720,0701,0691,0000,0829,0799,0787
22754676,0918,0940,0942,0921,0000,0830,0881,0862
22759684,0823,0819,0816,0811,0000,0830,0810,0799
22764688,0722,0698,0688,0691,0000,0829,0790,0782
22769692,0883,0897,0900,0890,0000,0830,0874,0861
22774696,0880,0892,0899,0896,0000,0830,0850,0848            
22779700,0706,0668,0659,0670,0007,0840,0778,0780
22784708,0823,0807,0798,0793,0000,0839,0880,0871
22789712,0935,0948,0939,0923,0008,0877,0884,0880
22794716,0759,0731,0711,0711,0004,0887,0812,0809
.
.
.
01143624,0916,0930,0925,0901,0000,0826,0864,0856
01148632,0783,0768,0756,0748,0000,0824,0781,0777

I tried to copy and paste the NULs, but it wouldn’t copy. In Notepad++ they look like “NUL” surrounded in the a black box. The blank line at the bottom would line 75521.

Here is my sketch:

#include <SdFat.h>
#include <SdFatUtil.h>
//
#define nb 10000UL
#define dt 5000
//
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
//
uint32_t bgnBlock, endBlock;
//
void setup(void)
{
  int ai[8];
//  
  int i;
  int j;
  int k;
  int l;
//  
  long m;
  long n;
  long o;
//  
  delay(2000);
//  
  pinMode(2, OUTPUT);
  digitalWrite(2, HIGH);
//
  !card.init();
  delay(500);
  !volume.init(card);
  delay(500);
  !root.openRoot(volume);
  delay(500);
  SdFile::remove(root, "RFC.CSV");
  delay(500);
  !file.createContiguous(root, "RFC.CSV", 512UL*nb);
  delay(500);
  !file.contiguousRange(bgnBlock, endBlock);
  delay(500);
//  
  for (j = 1; j <= 20; j++) {
    delay(250);
    digitalWrite(2, LOW);
    delay(250);
    digitalWrite(2, HIGH); 
  }
//  
  delay(500);
  uint8_t *pCache = volume.cacheClear();
  delay(500);
  memset(pCache, ' ', 512);
  delay(500);
//  
  for (i = 0; i <= 9; i++) {
   pCache[(50*i)+8] = ',';
   pCache[(50*i)+13] = ',';
   pCache[(50*i)+18] = ',';
   pCache[(50*i)+23] = ',';
   pCache[(50*i)+28] = ',';
   pCache[(50*i)+33] = ',';
   pCache[(50*i)+38] = ',';
   pCache[(50*i)+43] = ',';
   pCache[(50*i)+48] = '\r';
   pCache[(50*i)+49] = '\n';
  }
//  
  pCache[498]=' ';
  pCache[499]=' ';
//  
  pCache[510]='\r';
  pCache[511]='\n';
//  
  delay(500);
  !card.erase(bgnBlock, endBlock);
  delay(500); 
  !card.writeStart(bgnBlock, nb);
  delay(500);
  digitalWrite(2, LOW);
  delay(3000);
  digitalWrite(2, HIGH);
//  
  for (i = 0; i < nb; i++) {
    for (j = 0; j <= 9; j++) {
      m = micros();
      for (k = 0; k <=7; k++) {
        ai[k] = analogRead(k);
      }
//      
      n = m;
      for (k = 7; k >= 0; k--) {
        pCache[(50*j)+k] = m % 10 + '0';
        m /= 10;
      }
//    
      for (k = 0; k <= 7; k++) {
        o = ai[k];
        for (l = 3; l >= 0; l--) {
          pCache[(50*j)+(5*k)+9+l] = o % 10 + '0';
          o /= 10;
        }
      }
//      
      if (j < 9) {
        while (micros() < n + dt - 5) {
          delayMicroseconds(10);
        }
      }
    }
    !card.writeData(pCache);
//    
    while (micros() < n + dt - 5) {
      delayMicroseconds(10);
    }
  }
//  
  delay(500);
  !card.writeStop();
  delay(500);
  root.close();
  delay(500);
  file.close();
//  
  digitalWrite(2, LOW);
}
//
void loop(void) {}

Here is a picture of the frequency versus time:

http://www.flickr.com/photos/51887014@N02/4777501701/

I believe your problem is that the micros() result rolled-over in the middle of your 10 line loop. If that happens you will “WAIT” for a long time for micros() < n + dt - 5 to be true again…
(because n is still set to the LARGE result from the beginning of the loop.)

change the test to:
while ((micros()-n) < (dt-5))
{
delayMicroseconds(10);
}

However, It won’t delay on the roll-over at all.

You won’t get better than 200Hz because dt =5 milliseconds.
Why do you need that long? My accel doc states that the max read is 2ms…

CORRECTION:

The following statement should be:

if would take while (micros() < n + dt - 5) a long time to be false again…

I'm a little confused. I thought it took 70 minutes (or something like that) for micros to "roll over" back to zero. Are you talking about the roll over because I am only writing the last seven numbers to file??? This was all the space I had to still get 10 lines per block. Am I missing something??? It looks like the micros function is limited to an unsigned long.

I just happened to have dt set to 5000us in the code when I posted it. If I set it to 2000us I still only get 200-250 Hz. If I set it to 5000us I get a nice consistent 200 Hz which I prefer over a bouncy 200-250 Hz.

Not sure what a accel doc is???

Thanks, Andy

Yes, I thought you had written the entire time value in the file. So what I saw would represent a roll-over issue. Oh, you need to define m & n as unsigned longs , too. Otherwise you could run into issues at 35 minutes.

How long are you letting it run before you stop? It appears the code is creating a contiguous 5000kB file whether you fill it with data or not.

OK, I looked at the SDFat docs. You might want to put some checks for error returns in the code. I'd compare endBlock-bgnBlock to make sure you really got a contiguous file the size you want. I can assure you that just because a 2GB card appears to be empty, does NOT mean the file system will let you create say a 500MB contiguous chunk. So when you close the file and open the SD on a computer, is the filesize stated to be 3776kB or is it 5000kB?

Lewis

"Accel docs" is the datasheet for the accelerometer that I have. I assume some of the input data you are retrieving was from an accelerometer.

Hey man, you got a letter drought in your area or something. Giving you variables descriptive names that relate to the info they store will make your code much easier to debug when you run into problems, especially for other people to debug.

It doesn't affect the compiled code in any way, so there's no advantage in using one letter variables.

Anyways, on the point of hex files, you can download a hex plugin for Notepad++ here, http://sourceforge.net/projects/npp-plugins/files/, which will let you see exactly what bytes are being written to the card.

So, having used the hex plugin before, I can tell you that the 'NUL' relates to 0xFF, which generally means error. So it's writing a bunch of 'error' flags/bytes.

As ideasman says, try putting some of the error checking stuff back in, you can always comment it out once the code works.

The Cageybee

I will do put some of the error checking stuff back in the code see what it tells me. Any idea what it means when it dumps these 0xFF characters to the end of the file? Just a generic or default value?

BTW, loved the "letter drought" comment. I always get harassed for my non-descript variable names. Makes perfect sense to the person writing the code and no sense to everyone else. I'll try to change these as well.

Thanks, Andy

All,

I have been on travel for work. I took my arduino with me. While there, my uSD card died (I thought). So I went to RadioShck and bought a new one. It was seemingly the same thing. I put it in and interesting things happened. I ran it and it kindof worked. I told it to scan at 200hz for 30 minutes and although the LED lights never went off signifying it was done... I left it run overnight and in the morning I turned it off. Later, I checked the card and indeed the file was there and was complete. I checked the frequencies between scans and (other than the same one point dip early) it gave me 200 hz for 30 minutes.

I have tested it again today. I ran it for 5, 10, 15, 20, and 25 and the light went off and everything worked the way it is supposed to. I am currently testing the 30 minute again. I will repost the results.

Subsequently, I went back to the "dead" card and was able to revive it. However, it acts exactly the same as it did before.

Differences in the card: 1. The old card was formatted before I used it. The new one has never been formatted (by me). 2. The old card has a part number of SDSQ-002G-A11M and the new one has a part number of SDSQ-2048-A11M. I checked the Sandisk website and can't seem to find the -2048- one anywhere.

Something has to be different with the two cards. Any ideas??? Should I format the new card and see how it behaves??? Anyone have any other format ideas???

Thanks, Andy

I didn't know counterfeit cards were a problem. I got one of them from newegg and the other from RadioShack. The one from RadioSahck is the one that I can't seem to find anywhere on the Sandisk website. Also, the one that I can't find is the one that works better.

The card wrote data for 30 minutes successfully. I ran it for 60 minutes and it worked, but the time data got all jumbled after about 35 minutes. I am going to try it again.

Not really sure what happened, but it might be worth another shot.

Andy