next topic »

[on:]

Hi everybody,

This is my first post, so bear with me.

I'm building a disdrometer (raindrop counter) for a project, and I am in iffy shape. I've got a piezo disk attached to an Uno & have the SD breakout board from adafruit.

I have two questions for you guys; first, is it possible for me to regularly data with 10 kHz frequency with the Uno? I've been using
http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1208715493/11
to change the prescale factor and try and speed up the ADC, but when I put in a timer it still seems as though I'm not near 10kHz (I need to be able to see with about 100 us resolution, as a raindrop impact occurs over the span of 300us and I need to know its size).

Once I'm reading that fast, I need to have the data stored on a SD card. This may seem like an obvious question, but should I process the raw raindrop signal (just a voltage from the impact on the piezo disk) before I store it, or is there a way to store data at 10kHz? The issue that I face is that I might have to collect the signal data, process it, and then store it multiple times in a second during a high-intensity rain event. Writing to an SD card seems to take time.

These questions may sounds stupid, but I'm not good at programming and I need to build this sensor!

One final, stupid question - when I upload a sketch to Arduino (say the Datalogger example from SD), does it only start logging once I open the serial monitor?

[Reply #1 on:]

The man who asks his questions is dumb for a short time, he who don't ask his question is dumb forever - chinese verb (IIRC)

I have two questions for you guys

first, is it possible for me to regularly data with 10 kHz frequency with the Uno?

depens on the quality of the signal, standard analogRead() goes up to ~8500 readings per second.

tweaking with precision can give you ~50.000 readings per second.

Code:

// http://arduino.cc/forum/index.php/topic,56396.0.html
// and
// http://arduino.cc/forum/index.php/topic,74895.0.html

/*
  Sketch to demonstrate how to decrease time of analog read.
 it shortens the delay by manipulating registers and reducing the prescale
 timer value from 128 to 16;
 
 Note Serial Speed! Change to suit yourself!
 
 Sample Sketch From Arduino Cookbook by Margolis (O'Reilly Press)
 Modified by Willr March24, 2011
 
 Tested on Mega2560 Arduino board with an MA7361 Z output hooked to
 Sensor 5
 */

// CHANGE Sensor Pin to suit yourself...
const int sensorPin = 5;

// Change  number of entries to a value divisible by 100 only
const int numberOfEntries = 100;

unsigned long microseconds;
unsigned long duration1;
unsigned long duration2;

int results[numberOfEntries];

void setup()
{
  Serial.begin(9600); ///CHANGED
  Serial.flush();


  // standard analog read performance -- prescale == 128
  microseconds = micros();
  for (int i=0; i < numberOfEntries; i++)
  {
    results[i] = analogRead(sensorPin);
  }

  duration1 = micros() - microseconds;

  Serial.print(numberOfEntries);
  Serial.print("   readings took ");
  Serial.println(duration1);

  // This double loop just prints a neat table..
  for( int j=0; j < numberOfEntries; j=j+20)
  {
    for (int i=0; i < 20; i++)
    {
      Serial.print(results[j+i]);
      Serial.print(",  ");
    } //end loop i
    Serial.println();
  } //end loop j

  Serial.println();

  // prescale clock to 16
  bitClear(ADCSRA,ADPS0);
  bitClear(ADCSRA,ADPS1);
  bitSet(ADCSRA,ADPS2);

  // Performance with Changeed Prescale...
  microseconds = micros();
  for (int i=0; i < numberOfEntries; i++)
  {
    results[i] = analogRead(sensorPin);
  }

  duration2 = micros() - microseconds;
  Serial.print(numberOfEntries);
  Serial.print("   readings took ");
  Serial.println(duration2);

  for( int j=0; j < numberOfEntries; j=j+20)
  {
    for (int i=0; i < 20; i++)
    {
      Serial.print(results[j+i]);
      Serial.print(",  ");
    } //end loop i
    Serial.println();
  } //end loop j

  Serial.println();

  Serial.print("Ratio of read times is : ");
  Serial.println((float)duration1/duration2);
  Serial.println("********************");
}

void loop()
{
  // empty main loop is deliberate...
}

On my uno 100 readings were made in 1716 us => 18 us per reading. So the 300 us can give you 16 readings of 2 bytes = 32.  (programmers like powers of 2)

should I process the raw raindrop signal (just a voltage from the impact on the piezo disk) before I store it

No, you should capture the raw reading of 16 raindrops in RAM giving you a buffer of 512 bytes and write that to the SDcard (check the storage subforum for sdFastlib)
That said you probably want to include a timestamps (4 bytes) with every measurement making it 16 x 6 bytes = 96 bytes, So every 5 readings you should store to disk - and leave part of the 512byte buffer empty - that is really faster!

Processing the reading to whatever takes time, you probably not have. Better be ready for the next drop! Processing can be done afterwards. But it is your choice, what do you want. measure one drop, process the readings and display results || measure as many drops as possible.

or is there a way to store data at 10kHz?

SDfatlib can go beyond that but large chuncks is important -> read the threads

Writing to an SD card seems to take time.

Yes, some actions seem instantanious for us mere mortals but everything takes time writing to an SDcard is in the end just a physical process.

when I upload a sketch to Arduino (say the Datalogger example from SD), does it only start logging once I open the serial monitor?

No, it depends on your sketch. You can make it wait for a button/switch pressed at the end op setup() , or maybe better wait before the analogRead() is above a certain threshold. And in the same way you can make it stop.

Hope this helps,

[Reply #2 on:]

And please let us know the results of your study!

[Reply #3 on:]

thanks for the help! I definitely will!

another question - would it be faster to include a timestamp with every buffer that I fill rather than having a timestamp next to each recorded voltage?

[Reply #4 on:]

Yeah you can use one timestamp per 16 readings as all those readings are subsequent and belong to each other they make one logical unit,
I would propose to add two timestamps, one at the start and one at the end so you have a quite precise estimate of the time between teh readings.

So one record becomes:  <timestamp><reading1><reading2><reading3>...<reading15><reading16><timestamp2>

meaning 40 bytes for the measurement of one drop. 
The 512 byte buffer can now contain 480/40 = 12 readings.

[Reply #5 on:]

Okay. So I'm using the AnalogRead example from SDFat to store it, but I made some edits.

Two more questions. First, what does "cout" do? I've been searching for some documentation on cout and bout but cannot find any.

Second, is this the way to do a buffer? Should I be storing the values into a "buf" index before using bout/cout? Is bout just a buffer?

I'm timing it with microseconds and it looks like this will definitely be fast enough.

Code:

void loop() {
  uint32_t m;

unsigned long microseconds;
unsigned long duration;

  // use buffer stream to format line
  obufstream bout(buf, sizeof(buf));

  // start with time in millis
  bout << m;
for (int j=0; j<3; j++) { // collect 4 drops before logging
 
int g = analogRead(0); // check for positive signal

if (g>0) {
bout << g;
  microseconds = micros();
  // read analog pins and format data
 
   for (uint8_t ia = 0; ia < 15; ia++) {
   bout << ',' << analogRead(0);
  }
  duration = micros() - microseconds;
  bout << ', microseconds' << duration;
  bout << endl;
}
  // log data and flush to SD
  logfile << buf << flush;

  // check for error
  if (!logfile) error("write data failed");

#if ECHO_TO_SERIAL
  cout << buf;
#endif  // ECHO_TO_SERIAL

}

  bout << endl;
}

[Reply #6 on:]

bout << ',' << analogRead(0);

is the same as
Serial.print(bout);
Serial.print(',');
Serial.print(analogRead(0));
 etc

cout and cin stands for console out and console in; the << appends a variable to the output stream. cout and cin are defined in some stream library.

Many people like cout and cin for readablility,

[Reply #7 on:]

Okay. So my code is running slow - I have put a timer, and it's still taking about 100us per call when I log to SD.

Would removing the timer (the micros() lines) improve the rate? Right now I'm just storing 15 values at a time. The strategy is already little dubious.. the problem is that I only want to read voltages when a drop event happens, and I'm trying to get rid of low values that don't matter (the peak is all that really matters). Any ideas on speeding it up? To do what was suggested (10 or 15 drops with 16 data points each) do I just need a nested for loop before I output information?

Here's the code, still based on AnalogLogger from SDfat examples:

Code:

// datalogger
// uses RTClib from https://github.com/adafruit/RTClib
#include <SdFat.h>
#include <SdFatUtil.h>  // define FreeRam()
#include <Time.h>

#define CHIP_SELECT SS_PIN  // SD chip select pin
#define USE_DS1307       0  // set nonzero to use DS1307 RTC
#define SENSOR_COUNT     1  // number of analog pins to log
#define ECHO_TO_SERIAL   1  // echo data to serial port if nonzero


// file system object
SdFat sd;

// text file for logging
ofstream logfile;

// Serial print stream
ArduinoOutStream cout(Serial);

// buffer to format data - makes it eaiser to echo to Serial
char buf[80];
//------------------------------------------------------------------------------
#if SENSOR_COUNT > 6
#error SENSOR_COUNT too large
#endif  // SENSOR_COUNT
//------------------------------------------------------------------------------
// store error strings in flash to save RAM
#define error(s) sd.errorHalt_P(PSTR(s))
//------------------------------------------------------------------------------

//------------------------------------------------------------------------------
void setup() {
 
bitClear(ADCSRA,ADPS0);
bitClear(ADCSRA,ADPS1);
bitSet(ADCSRA,ADPS2);


  Serial.begin(9600);
  Serial.flush();
  // pstr stores strings in flash to save RAM
  cout << endl << pstr("FreeRam: ") << FreeRam() << endl;

#if WAIT_TO_START
  cout << pstr("Type any character to start\n");
  while (Serial.read() < 0) {}
#endif  // WAIT_TO_START



  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  // if SD chip select is not SS, the second argument to init is CS pin number
  if (!sd.init(SPI_HALF_SPEED, CHIP_SELECT)) sd.initErrorHalt();

  // create a new file in root, the current working directory
  char name[] = "LOGGER00.CSV";

  for (uint8_t i = 0; i < 100; i++) {
    name[6] = i/10 + '0';
    name[7] = i%10 + '0';
    if (sd.exists(name)) continue;
    logfile.open(name);
    break;
  }
  if (!logfile.is_open()) error("file.open");

  cout << pstr("Logging to: ") << name << endl;

  // format header in buffer
  obufstream bout(buf, sizeof(buf));

  bout << pstr("sens, ");
  bout << pstr("microsecond, ");


#if ECHO_TO_SERIAL
  cout << buf << endl;
#endif  // ECHO_TO_SERIAL
}
//------------------------------------------------------------------------------
void loop() {
  uint32_t m;
while(true) {

  int record;
  // use buffer stream to format line
  obufstream bout(buf, sizeof(buf));
  int i = 0;
if (analogRead(0)>5) {
  bout << endl;
  // take a timestamp before the data is logged

  // take 15 values when it's positive
  record = micros();
for (uint8_t ia = 0; ia < 15; ia++) {

bout << analogRead(0);
bout << endl;
  }
int timeelapsed = micros()-record;
bout << "Time Elapsed, us" << "," << timeelapsed;
bout << endl;
}
// log data and flush to SD
logfile << buf << flush;

// check for error
  if (!logfile) error("write data failed");

#if ECHO_TO_SERIAL
  cout << buf;
#endif  // ECHO_TO_SERIAL

  bout << endl;
}
}

[Reply #8 on:]

Some points:

- Serial.begin(115200); iso 9600  is 12x as fast for al serial io
- "Time Elapsed, us" is quite a long string taht adds no value only time loss. "T:" would be sufficient, would free 10+ places for analog samples.
- record should be unsigned long, that is the return type of micros();
- Furthermore you also write buf to the logfile if there is no data (analogRead() >5)
- lots of obselete code ..
- Restyling your code makes it more readable (CTRL-T in the IDE does this automatically)

Code:

void loop()
{
  uint32_t m;
  while(true)
  {
    int record;
    // use buffer stream to format line
    obufstream bout(buf, sizeof(buf));
    int i = 0;
    if (analogRead(0)>5)
    {
      bout << endl;
      // take a timestamp before the data is logged

      // take 15 values when it's positive
      record = micros();
      for (uint8_t ia = 0; ia < 15; ia++)
      {
        bout << analogRead(0);
        bout << endl;
      }
      int timeelapsed = micros()-record;
      bout << "Time Elapsed, us" << "," << timeelapsed;
      bout << endl;
    }
    // log data and flush to SD
    logfile << buf << flush;
    // check for error
    if (!logfile) error("write data failed");

#if ECHO_TO_SERIAL
    cout << buf;
#endif  // ECHO_TO_SERIAL

    bout << endl;
  }
}

Just a quick refactor of your loop to minimize code,
changed all readings to one line separated by "," so Excel can read it easily. Last field is the timeelapsed.

Code:

void loop()
{
  obufstream bout(buf, sizeof(buf));
  if (analogRead(0) > 5)
  {
    unsigned long start = micros();
    for (uint8_t ia = 0; ia < 15; ia++)
    {
      bout << analogRead(0) << "," ;
    }
    unsigned long timeElapsed = micros() - start;
    bout << timeElapsed << endl;

    logfile << buf << flush;
    if (!logfile) error("write data failed");
  }
}

Just picking the peaks > 5  is a bit more work (not optimized for speed)

Code:

setup quite similar ..

int peak[20];
unsigned long time[20]
int idx = 0;

void loop()
{
  peak[idx] = analogRead(0);
  time[idx] = micros();
  if (peak[idx] > 5)
  {
    idx++;
    if (idx == 20)  // if we have 20 readings we write them to SDcard
    {
      idx = 0;
      for (int i=0; i<20; i++)
      {
        logfile << time[i] << "," << peak[i] << endl;
      }
    }
    logfile << flush;
    if (!logfile) error("write data failed");
  }
}

Give it a try

[Reply #9 on:]

Thanks Rob - that works, but for some reason it's running slower than my older (and messier code).

Is it because a real buffer is not being used correctly? I've attached a picture of my raindrop records - it seems like the initial part of the code eats up time (finding out whether or not analogRead has a positive value), and by the time the data is set to record it the signal from the raindrop has already peaked... any thoughts?



(the numbers are somewhat meaningless). Right now it's taking me about 2500 us to read a drop --> 170 us resolution.

thoughts about speeding it up? catching the first values of a raindrop event (but not all of the zeroes between in between?)

Before I was doing something like this inside my loop()

Code:

h = analogRead(0);
if (h>0) {
...
}

And print the h value, followed by all of the other positive analog values.


thanks.

[Reply #10 on:]

Think that writing to the SDcard is a bottleneck as these work with 512bytes sectors, and depending on the card type you can have a serious delay. Removing the timestamps and , and CRLF and writing the data in binary format would increase the speed. Check for the fast IO code in the storage section of fat16lib.