Pages: 1 [2] 3   Go Down
Author Topic: Scanning Analog Inputs and Writing to uSD Card  (Read 3450 times)
0 Members and 1 Guest are viewing this topic.
0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks Cageybee!!!  I always forget to take advantage of the Serial stuff.  If I read your reply correctly, it would be the best to write in the raw data format to the SD card to be able to scan fast and consistently.  Once I have the data on the card I could load a second sketch (or have another arduino ready) and print all that data out to the serial port while capturing it with VB or something (or I could write it to another card).  Anyone have a link to a code that will read the serial stuff and put it in a file for VB6.  BTW I know that VB6 is an old code, but I still use it a ton.

I am going to play around with some other things today, but may give this a shot over the weekend.

Thanks,
Andy
Logged

Fife, Scotland
Offline Offline
Newbie
*
Karma: 0
Posts: 40
Electronicoid Android Electronics Calculator
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

If you're going to be processing the data using VB6 you could just use hyperterminal, or any other terminal software to just capture the serial data to a file, then use VB6 to parse and split the data up in to meaningful data.

There are a few things you need to think about first though, namely how you want the data to be stored and whether you want a timestamp for each pass of samples.

Basically, what's your data structure going to look like.

So, say you wanted to store the 8 analog inputs values as 'int's and have a timestamp generated by the millis() function, which is an 'unsigned long', each loop will be: (8 * 2bytes) + 4bytes. So that's 20 bytes total.

Now, sdcards are split into 512byte blocks, so you'd have to use a buffer to store the data, then once the data's full, write the data to the card, increment a counter to write the next block to, then start overwriting the buffer, till it's full, and so on, and so on.

To simplify things a little you'd probably be better off writing the data to the card when you've got 500bytes of data and just leaving the final 12bytes as 'junk'. So that means you'll be storing the data from 25 'loops' to the card each time.

To be honest, I've never written to sd in RAW before, but if you give me a better idea of what you want your datastructure to look like, I'll look into it and come up with some code you can use/adapt.

Regards,
The Cageybee
« Last Edit: July 02, 2010, 11:40:41 am by cageybeeuk » Logged

Fife, Scotland
Offline Offline
Newbie
*
Karma: 0
Posts: 40
Electronicoid Android Electronics Calculator
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Okay. So I've had a little look see into this and think I've come up with something.

There are a couple of RAW SD libraries available, but I couldn't get one of them to compile. A few people are having the same problem with it, but no solutions of how to resolve it yet.

I did get the one from here http://sourceforge.net/projects/arduinoweather/files/sdcardlibrary.zip/download to compile though.

Assuming you wanted the data structure I outlined above, you could use the following code to write the data to the card.

Quote
#include <SDCARD.h>

volatile unsigned char buffer[512];  // Read/Write buffer
unsigned char bufferPos = 0;  // Current position in the buffer
unsigned long sector = 0;  // Current sector
int error = 0;   // Error flag

void setup(){
  Serial.begin(9600);
}

void loop(){
  sampleInputs();
}

void sampleInputs(){
  int sampleData;        // Raw sampled data
  byte loByte, hiByte;   // Hold raw data split into hi & lo bytes
  for (int i = 0; i < 8; i++){    // Sample all inputs, spilt into hi & lo bytes, push to buffer
    sampleData = analogRead(i);
    loByte = sampleData &0xFF;
    hiByte = sampleData >>8;
    buffer[bufferPos] = loByte;
    bufferPos++;
    buffer[bufferPos] = hiByte;
    bufferPos++;
  }
  unsigned long timeStamp = millis();  
  buffer[bufferPos] = timeStamp &0xFF;  // Split timestamp up into 4 bytes and push to buffer
  bufferPos++;
  buffer[bufferPos] = timeStamp >>8;
  bufferPos++;
  buffer[bufferPos] = timeStamp >>16;
  bufferPos++;
  buffer[bufferPos] = timeStamp >>24;
  bufferPos++;
  
  if (bufferPos >=499) {writeToSD();}  // If buffer full, write it to the SDcard
}


void writeToSD(){   // Writes data to card, increments sector, resets buffer
  error = SDCARD.writeblock(sector);  // Write data in the buffer to the card at current sector
  if (error !=0){  // If an error occured
    Serial.print("sd card write error... code =  ");
    Serial.println(error);      
  }
  else {  // If write was successful
    sector++;   // Increment current sector num
    bufferPos = 0;  // Reset buffer position ready to overwrite
  } 
}


Then when it comes to reading the data back, you can reassemble the data, put in in a nice format and spit it out over serial. Say you wanted to save it as a CSV file, you can just format the data in that format as you send it over serial and just capture the data to a file using hyperterminal or similar.

I've got a couple of ideas of things you could do.

If you've got a free digital pin you could have a sliding switch that selects whether you want to write data to the card or dump the data over serial.

Also, if you were to store the sector position to EEPROM, between launches you could turn the logger off, then when you're ready to launch, turn the logger on, the current sector is loaded from EEPROM and the data continues to be written from the last sector so you don't overwrite your data. You could have the logger write 1 sector of 0xFF's to the card on startup so you can tell when a new logging session has started.

Are you using a GPS? Or are you just making analog samples?

Let me know if you want me to go any further with this.

Regards,
The Cageybee
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Cageybee,

You are getting way ahead of me.  I've got some catching up to do.  I'll do my best to keep up.

A little more testing of the old way revealed that is it not a buffer issue, I think.  I put a delay of 20ms inside the loop.  The spikes were further apart but occured after the same amount of data was written to the file as with no delay.  If it was a buffering thing this would get better the slower and slower I tried to write things to the file, right???

I am using the SDFat library and it has a raw write example included with it.  I will try to get it working and see what happens.  I will also take the time look through your code and make it makes sense to me.

Crowley,

I was also thinking about ways to trigger things to start recording.  The problem is once something is detected, I want to start recording slightly earlier.  In rockets things happen extremely fast.  The GPS modules that I have seen all run at very low frequencies (~10 Hz).  Before the GPS detects that the rocket moved, I could be half way to motor burn-out.

My father-in-law (a pilot) has a saying that I can use here.  There is no such thing as too much runway.  In this case, there is no such thing as too much data.  I would rather have 70 hours of data with 10 useful seconds than 8 seconds of data with 10 seconds of useful data.  Data is easier to delete than generate.

Thanks again for everyone's input and help,
Andy
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 19
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

At what amount of data does the spike occur?  BTW, it's the amount that causes the physical write to take place, rarely time.
The recent suggestions about storing data and writing out 500 byte blocks IS buffering -- internally by your program.
I'm guessing ,but are the spikes at .5k, 4k or 8k ?  
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I am assuming 20 bytes are written each loop.  8*2 for the integers plus 4 for the unsigned long.  There are really two different delays in writting the data.  The first one occurs every 12 or 13 loops and are ~0.05 seconds apart.  This one has a time step of ~0.007 seconds and is not too big of a deal.  12 or 13 loops give me 240 and 260 bytes between the small delays.

The large delays occur every ~13100 loops and this delay is 0.2-0.4 seconds long.  This one is my major concern.  13100 loops gives 262 kB between long delays and 56-57 seconds.

It appears to me that I have a factor of two missing somewhere.  If I multiply the number of bytes by a factor of two I get:
1. 480-520 bytes between small jumps
2. 524 kilobytes between big jumps
These are both dangerously close to 512 bytes and 512 kB.  

My questions are:
1. Is there a factor of two that I am missing somewhere???
2. Is there a fix for this problem???
     a. Writing a raw data file???
     b. Writing in blocks of 512 bytes instead of small increments???
3. Why did this problem not go away when I put a 20ms delay in each loop???

Thanks,
Andy
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I found the factor of two.  The code does not write things as integers, it writes everything as uint8_t.  The time is 8 or 9 bytes, each analog is 1 to 4 bytes, and the strings are 11 bytes.  The total for my case is either 41 or 42 bytes per loop.

I still have the other questions.

Thanks,
Andt
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If the rocket goes out of range of the telemetry or telemetry is lost, I will not have the GPS data.  It also can be used as a crude tracking device.  If for some reason the rocket gets absolutely lost, you could just drive around until you get close enough to the rocket to acquire a signal again.
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This may be some very good news.  I looked into the rawrite example with the SDFat libraries.  It compiled and ran smoothly.  If what the sketch tells you is true this would be impressive.  It says that it it writing 51200 bytes of data per second for 100 seconds.

So if this alone is true, it is something great.

Then just for curiosity sake, I put the card in my computer and it has an ascii readable file.  It is 5000KB in size and took 100 seconds which is 50000 bytes per second.  If I can get this data rate out, I will have over 1200 hz or 0.83ms time step.  Right now I cannot successfully get 4ms to work.

I will look into it more when I get time, but this looks very promising!!!
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 19
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Does that include the reading of the 8 analog devices, too?
I wrote a sample read that uses pointers.  It takes 900 usec to read and store the 8 values in the buffer (NOT WRITE to SD), using 20 bytes binary.

I haven't tried to time the sampleInputs() listed in this thread.
(I think mine would be faster -- It has no loops for creating ASCII data.)
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

That is fast!!!  I think writing to the SD card appears to be a bigger bottleneck than scanning.  In my old code, if I had it just write out meaningless numbers instead of scanning it was around 700 us faster per loop.

This will allow me to do other stuff on top of scanning the analog channels while still getting a high enough frequency.
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well, I have made some progress towards this.  I have a sketch that writes time (microseconds) and all 8 analog channels.  I can get about 200 Hz out of it semi-consistently.

Here are the few issues I am having now:

1.  I can get 200 Hz out of it with the exception of one point at the very front of the file that drops to 50-150 Hz.  It is always one point and always the 31st line.  I am doing 10 lines per 512 byte block so this would be the first line of the third block.  I suppose it could be the writing of the second block that really causes this.  Why does it do it there and nowhere else???

2.  I can only seem to get 75520 good lines out of this sketch.  I'm not really sure why.  Line 75521 contains a whole bunch of "NUL" as viewed in Notepad++.  Any suggestions on getting over this hump???  The files I am generating are 5000KB in size and the card is 2GB so I know I am not running out of space.  After 3866624 everything goes to NUL???

I also noticed, I can only get 250-200 Hz maximum out of it.  200 for line 0-8 and 250 for line 9.  It still has the one point dip in the front.

Thanks,
Andy
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Just confirmed the number of "NUL" characters is equal to the remaining characters that should have been written to the file.
Logged

Fife, Scotland
Offline Offline
Newbie
*
Karma: 0
Posts: 40
Electronicoid Android Electronics Calculator
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Any chance you could post the current sketch and some of the file that's stored on the sdcard.

A few lines of 'good' recordings and the bad ones.

Would be even more useful if you could post them as read by a hex editor, as then it's easier to see exactly what bytes have been written to the card.

The Cageybee
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 25
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Here is a truncated file (not really code):

Code:
22679564,0480,0417,0407,0317,0000,0830,0576,0513
22684576,0558,0547,0550,0497,0000,0831,0761,0687
22689580,0724,0722,0720,0675,0000,0830,0834,0779
22694592,0622,0568,0539,0511,0000,0830,0768,0739
22699596,0681,0631,0593,0557,0000,0829,0801,0774
22704596,0890,0895,0881,0836,0000,0830,0883,0858
22709600,0814,0800,0786,0760,0000,0830,0813,0799
22714612,0718,0688,0671,0660,0000,0829,0791,0783
22719620,0880,0893,0892,0873,0000,0830,0874,0862
22724628,0881,0893,0898,0889,0000,0830,0850,0850            
22729636,0702,0667,0653,0659,0000,0829,0767,0775
22734644,0798,0788,0778,0770,0000,0830,0836,0835
22739652,0902,0914,0911,0894,0000,0830,0858,0854
22744660,0738,0710,0694,0692,0000,0829,0772,0773
22749668,0744,0720,0701,0691,0000,0829,0799,0787
22754676,0918,0940,0942,0921,0000,0830,0881,0862
22759684,0823,0819,0816,0811,0000,0830,0810,0799
22764688,0722,0698,0688,0691,0000,0829,0790,0782
22769692,0883,0897,0900,0890,0000,0830,0874,0861
22774696,0880,0892,0899,0896,0000,0830,0850,0848            
22779700,0706,0668,0659,0670,0007,0840,0778,0780
22784708,0823,0807,0798,0793,0000,0839,0880,0871
22789712,0935,0948,0939,0923,0008,0877,0884,0880
22794716,0759,0731,0711,0711,0004,0887,0812,0809
.
.
.
01143624,0916,0930,0925,0901,0000,0826,0864,0856
01148632,0783,0768,0756,0748,0000,0824,0781,0777            
I tried to copy and paste the NULs, but it wouldn't copy.  In Notepad++ they look like "NUL" surrounded in the a black box.  The blank line at the bottom would line 75521.

Here is my sketch:

Code:
#include <SdFat.h>
#include <SdFatUtil.h>
//
#define nb 10000UL
#define dt 5000
//
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;
//
uint32_t bgnBlock, endBlock;
//
void setup(void)
{
  int ai[8];
//  
  int i;
  int j;
  int k;
  int l;
//  
  long m;
  long n;
  long o;
//  
  delay(2000);
//  
  pinMode(2, OUTPUT);
  digitalWrite(2, HIGH);
//
  !card.init();
  delay(500);
  !volume.init(card);
  delay(500);
  !root.openRoot(volume);
  delay(500);
  SdFile::remove(root, "RFC.CSV");
  delay(500);
  !file.createContiguous(root, "RFC.CSV", 512UL*nb);
  delay(500);
  !file.contiguousRange(bgnBlock, endBlock);
  delay(500);
//  
  for (j = 1; j <= 20; j++) {
    delay(250);
    digitalWrite(2, LOW);
    delay(250);
    digitalWrite(2, HIGH);
  }
//  
  delay(500);
  uint8_t *pCache = volume.cacheClear();
  delay(500);
  memset(pCache, ' ', 512);
  delay(500);
//  
  for (i = 0; i <= 9; i++) {
   pCache[(50*i)+8] = ',';
   pCache[(50*i)+13] = ',';
   pCache[(50*i)+18] = ',';
   pCache[(50*i)+23] = ',';
   pCache[(50*i)+28] = ',';
   pCache[(50*i)+33] = ',';
   pCache[(50*i)+38] = ',';
   pCache[(50*i)+43] = ',';
   pCache[(50*i)+48] = '\r';
   pCache[(50*i)+49] = '\n';
  }
//  
  pCache[498]=' ';
  pCache[499]=' ';
//  
  pCache[510]='\r';
  pCache[511]='\n';
//  
  delay(500);
  !card.erase(bgnBlock, endBlock);
  delay(500);
  !card.writeStart(bgnBlock, nb);
  delay(500);
  digitalWrite(2, LOW);
  delay(3000);
  digitalWrite(2, HIGH);
//  
  for (i = 0; i < nb; i++) {
    for (j = 0; j <= 9; j++) {
      m = micros();
      for (k = 0; k <=7; k++) {
        ai[k] = analogRead(k);
      }
//      
      n = m;
      for (k = 7; k >= 0; k--) {
        pCache[(50*j)+k] = m % 10 + '0';
        m /= 10;
      }
//    
      for (k = 0; k <= 7; k++) {
        o = ai[k];
        for (l = 3; l >= 0; l--) {
          pCache[(50*j)+(5*k)+9+l] = o % 10 + '0';
          o /= 10;
        }
      }
//      
      if (j < 9) {
        while (micros() < n + dt - 5) {
          delayMicroseconds(10);
        }
      }
    }
    !card.writeData(pCache);
//    
    while (micros() < n + dt - 5) {
      delayMicroseconds(10);
    }
  }
//  
  delay(500);
  !card.writeStop();
  delay(500);
  root.close();
  delay(500);
  file.close();
//  
  digitalWrite(2, LOW);
}
//
void loop(void) {}
Logged

Pages: 1 [2] 3   Go Up
Jump to: