Go Down

Topic: Aquisition rate with SD Card (Read 2674 times) previous topic - next topic

diogotec

#30
Jul 25, 2017, 09:24 pm Last Edit: Jul 26, 2017, 10:22 am by diogotec
For what I know of working with bio-data, the first acquisition is at t=0!
But you are more than right! I have checked with data that I had on my pc and at t=1 second (and starting at t=0s) we have the frequency acquisitions + 1 :)
It's all correct, thanks @DrDooom!

Quote
Ur last Question about the overflowing:
The two functions millis() and micros() work with the largest possible number "unsigned long". An unsigned long goes back to 0 when it overflows. This also happens with millis after ~50 days and micros after ~70 minutes.
When an unsigned long e.g. Is calculated: 4294967280 + 120, the variable contains the value 105. I would therefore always calculate the time for the next execution. This can then overflow but always fits to the overflowed millis() or micros().

There can only be a Problem when next_aq_time = 4294967295; (last micro befor overflow) and something in your code needs some micros and the in the next loop micros has already overflown (micros returns eg 0)!
About the overflow of the micros() I can't actually predict or imagine when I will have a problem!
Even if I have next_aq_time = 4294967295, won't micros() be always less than it until it reaches the cycle again and then next_aq_time overflows and goes back to 0 but also does micros()?
EDIT: Just ran the code for 80 minutes and I got 1600000. Was expecting 1600001, maybe I lost one during the micros() overflow? OR could be because I used millis() to count the stop time and it has less precision :)
Will test now for 2 overflows and see if I lose one more acquisition.

Quote
There are also hidden class and functions to read low level sd card data. On this level you have to worry about the fragmentation of files on the sd card. :-(
I've forgotten to ask about this. What do you mean?

DrDooom

Quote
For what I know of working with bio-data, the first acquisition is at t=0!
Well, u can change the code / order of events to handle it exactly as desired with first acqusition at t=0 and a working frequency calculation... I would change the code to run allways 3ms interval:
Code: [Select]
const long SERIAL_BAUD  = 9600;//Serial

bool          acquireSensorRequestStop = false;
bool          acquireSensorActive = false;
unsigned long acquireSensorDataNextMicros = 0;
unsigned long acquireSensorDataDelayMicros = 3000;
unsigned long acquireSensorCount = 0;
unsigned long acquireSensorStartMicros = 0;
unsigned long acquireSensorEndMicros = 0;

void setup() {
  Serial.begin(SERIAL_BAUD);
  delay(500);
  while (!Serial);//Warten bis eine Serielle Verbindung kommt!
  Serial.println("acquirement started");
  acquireSensorRequestStop = false;
  acquireSensorActive = true;
  acquireSensorDataNextMicros = micros();
  acquireSensorStartMicros = micros();
}

void loop() {
  if(acquireSensorActive){
    if(micros()>=acquireSensorDataNextMicros){
      if(acquireSensorRequestStop){
        acquireSensorEndMicros = micros();
        acquireSensorRequestStop = false;
        acquireSensorActive = false;
       
        Serial.println("acquirement stopped");
        Serial.println("acquireSensor Duration:  " + (String)(acquireSensorEndMicros-acquireSensorStartMicros) + " micros (" + (String)((acquireSensorEndMicros-acquireSensorStartMicros)/1000.0) + " ms) -> should fit to 3 ms intervall");
        Serial.println("acquireSensor Count:     " + (String)acquireSensorCount + " x");
        Serial.print  ("acquireSensor Frequency: ");Serial.print((acquireSensorCount / (float)(acquireSensorEndMicros-acquireSensorStartMicros)) * 1000.0 * 1000.0, 3); Serial.println(" Hz");
      } else {
        acquireSensorDataNextMicros += acquireSensorDataDelayMicros;
        //Ecquire Sensor Data...
        acquireSensorCount++;
      }
    } else if(micros()-acquireSensorStartMicros > 1000 * 1000 * 5){ acquireSensorRequestStop = true; }//Dont stop immediately, only request to stop!
  }
}

The last line with acquireSensorRequestStop = true; is important. It does not stop the acquirement, it only requests the stop. The 3ms interval checks if the stop is requested (if(acquireSensorRequestStop){) and stops or acquires more sensor data.

With different durations / stop requests i got this results:
Code: [Select]

request to stop after 1 ms
acquirement started
acquirement stopped
acquireSensor Duration:  3005 micros (3.00 ms) -> should fit to 3 ms intervall
acquireSensor Count:     1 x
acquireSensor Frequency: 332.779 Hz

request to stop after 10 ms
acquirement started
acquirement stopped
acquireSensor Duration:  12001 micros (12.00 ms) -> should fit to 3 ms intervall
acquireSensor Count:     4 x
acquireSensor Frequency: 333.306 Hz

request to stop after 1000 ms
acquirement started
acquirement stopped
acquireSensor Duration:  1002001 micros (1002.00 ms) -> should fit to 3 ms intervall
acquireSensor Count:     334 x
acquireSensor Frequency: 333.333 Hz

request to stop after 5000 ms
acquirement started
acquirement stopped
acquireSensor Duration:  5001002 micros (5001.00 ms) -> should fit to 3 ms intervall
acquireSensor Count:     1667 x
acquireSensor Frequency: 333.333 Hz

request to stop after 60000 ms
acquirement started
acquirement stopped
acquireSensor Duration:  60003004 micros (60003.00 ms) -> should fit to 3 ms intervall
acquireSensor Count:     20001 x
acquireSensor Frequency: 333.333 Hz

Well, with low durations like < 1000 ms it can be difficult to achieve exactly 333,333 hz.

Maybe u should use an Arduino Due with 84Mhz. I have one, but only tested the code on Arduino MkrZero with 48Mhz.

Quote
About the overflow of the micros() I can't actually predict or imagine when I will have a problem!
I think this is the biggest problem. If the measurements with some users simply do not continue after 70/140/210 ... minutes!
The Problem can happen, if...
next_aq_time = 4294967295;
and
micros() is 4294967294 (~1 less than next_aq_time)
if the Arduino is doint something longer than 2 micros and is blocked, the next micros() will overflow to 0.
So
if(millis()>=next_aq_time){
will not be executed and next_aq_time never overflows because there is no next_aq_time += 3000;.


Quote
I've forgotten to ask about this. What do you mean?
I would not work with the hidden functions. They can be removed in updates or no longer work as before.

diogotec

Thanks for the code :) that issue is also solved now!

About the overflowing: Yeah, I see! Is there any way we can put safeguards?

DrDooom

What would be your approach?

diogotec

Something like

if((micros()>=acquireSensorDataNextMicros) OR (next_aq_time == 4294967295 AND micros() <=1))

DrDooom

Hmm, I think it could still come to problems. Perhaps you could try that out?
Code: [Select]

if(micros()>=acquireSensorDataNextMicros || micros() < acquireSensorDataNextMicros - acquireSensorDataDelayMicros){//time >= next run or time is unexpectedly to small

diogotec

#36
Jul 26, 2017, 04:15 pm Last Edit: Jul 26, 2017, 04:20 pm by diogotec
How can I test such simulation? It is unlikely to get into that situation, so test it is hard!

christop

The recommend way to test for an expired interval is like this:

Code: [Select]
if (millis() - lastMillis >= interval) {
    lastMillis += interval;
    // your code here
}


(That can easily be changed to use micros().)

Just keep track of the last time your event expired, and then subtract it from the current time and see if that's reached your interval. This technique is immune to millis()/micros() wrap-around.

diogotec

Hey @christop,
That way was similar to my first attempt. For some reason I always get 1 or 2 less acquisitions than expected.
By the way, for working at overflow shouln't I do abs(millis() - lastMillis)?

Didn't understood tho I can use that code to test the overflow case!

christop

Hey @christop,
That way was similar to my first attempt. For some reason I always get 1 or 2 less acquisitions than expected.
By the way, for working at overflow shouln't I do abs(millis() - lastMillis)?

Didn't understood tho I can use that code to test the overflow case!
Nope, millis() - lastMillis always gives an unsigned long value (both millis() and lastMillis must be unsigned long), so it works out correctly even in the presence of overflow/wraparound.

One way to force it to "expire" immediately (to take an acquisition right away) is to initialize lastMillis to millis() - interval. I'm not sure if that would fix the exact problem you're seeing, though.

diogotec

#40
Jul 27, 2017, 02:16 am Last Edit: Jul 27, 2017, 09:05 am by diogotec
@DrDooom,

Altough I don't know how to test I am pretty sure that that wrap in the if() you suggested should work for the case of acquireSensorDataNextMicros = overflow value and the micros() go to beyond overflow.
But what about the case that we have micros = 9402904924 (random value but near overflow) and acquireSensorDataNextMicros = 2. In that case 94029904924 will be higher than acquireSensorDataNextMicros and it will cause one acquisition out of tempo! I think although I got correct values at the end of time, one of them wasn't spaced 3ms from the previous

1st approach:

if(
(micros()>=acquireSensorDataNextMicros) && (acquireSensorDataNextMicros - acquireSensorDataDelayMicros <=  acquireSensorDataNextMicros))
||
micros() < acquireSensorDataNextMicros - acquireSensorDataDelayMicros)

but this will not work at the first acquisition.
So I came out with:
if((acquisitions == 0 && micros() >= acquireSensorDataNextMicros) || ((micros()>=acquireSensorDataNextMicros) && (acquireSensorDataNextMicros - acquireSensorDataDelayMicros <= acquireSensorDataNextMicros) ) || (micros() < acquireSensorDataNextMicros - acquireSensorDataDelayMicros)) { //time >= next run or time is unexpectedly to small


What you do you think?
Have tested it for 1 second, 3 seconds, 6 seconds, 9 seconds, 5 minutes, and the obtained values were correct. Will now test for 293 minutes to see how it beaves in overflow.
EDIT: it failed! Got millions of more acquisitions than expected

cattledog

#41
Jul 27, 2017, 04:43 am Last Edit: Jul 27, 2017, 09:46 am by cattledog
Quote
Altough I don't know how to test I am pretty sure that that wrap in the if() you suggested should work for the case of acquireSensorDataNextMicros = overflow value and the micros() go to beyond overflow.
But what about the case that we have micros = 9402904924 (random value but near overflow) and acquireSensorDataNextMicros = 2. In that case 94029904924 will be higher than acquireSensorDataNextMicros and it will cause one acquisition out of tempo! I think although I got correct values at the end of time, one of them wasn't spaced 3ms from the previous
The way to handle roll over is with the subtraction of unsigned values. You always need to subtract the previous value from the current value to find the interval. It will always be a positive number. Google twos complement if you want the explanation of why the math works out this way.  If you want something to happen every 3000 microseconds, the construct needs to be
Code: [Select]
if(currentMicros - previousMicros >= 3000)//this is unaffected by rollover
{//do something}


You actually had this formulation earlier in the thread, and moved away from it for some reason I didn't understand.

See this example of subtraction with unsigned values where a large value(last reading) is subtracted from a small one (current reading after rollover).
Code: [Select]
byte currentX;
byte previousX;
unsigned int currentY;
unsigned int previousY;
unsigned long currentZ;
unsigned long previousZ;
unsigned long lastMicros;
unsigned long acquireSensorDataNextMicros;

void setup() {
  Serial.begin(115200);
  previousX = 255;//before rollover
  currentX = 1;//after rollover
  byte valX = (currentX - previousX);
  Serial.println(valX);
  previousY = 65535;//before rollove
  currentY = 1;//after rollover
  unsigned int valY = (currentY - previousY);
  Serial.println(valY);
  previousZ = 4294967295;//before rollover
  currentZ = 1;//after rollover
  unsigned int valZ = (currentZ - previousZ);
  Serial.println(valZ);
  lastMicros = 9402904924;// ("random value but near overflow" ?? )
  acquireSensorDataNextMicros =2;
  unsigned long valMicros = (acquireSensorDataNextMicros - lastMicros);
  Serial.println(valMicros);

}

void loop(){}

diogotec

Quote
You actually had this formulation earlier in the thread, and moved away from it for some reason I didn't understand.
Because while testing I didnt get the right acquisitions as in the code I am using now

DrDooom

#43
Jul 27, 2017, 09:34 am Last Edit: Jul 27, 2017, 10:24 am by DrDooom
Well, once again like my prev post:
Quote
The Problem can happen, if...
acquireSensorDataNextMicros = 4294967295;
and
micros() is 4294967294 (~1 less than next_aq_time)
if the Arduino is doint something longer than 2 micros and is blocked, the next micros() will overflow to 0.
So
if(millis()>=acquireSensorDataNextMicros){
will not be executed and acquireSensorDataNextMicros never overflows because there is no acquireSensorDataNextMicros += 3000;
Code: [Select]

if(micros()>=acquireSensorDataNextMicros || micros() < acquireSensorDataNextMicros - acquireSensorDataDelayMicros){//time >= next run or time is unexpectedly to small


I have tested the behavior by writing a function _millis() which is just before the overflow. So dont need to wait 70 days ^^:
Code: [Select]

unsigned long _millis(){
  return millis() + 4294965000;//max 4294967295
}


Then I wrote a small code and used _millis() instead of millis(). The code tries to keep a 1 ms intervall and a counter++;.
With an if(acquireSensorDataNextMicros==4294967295){ delay(2); } to simulate a task with 2ms block time.

And it is actually that acquireSensorDataNextMicros does not overflow. With the above/extended IF there are no problems. I would recommend you to test the behavior as it is especially important for your project.
I already had different projects, in which a problem only occurs irregularly and it is very difficult to analyze this later.

diogotec

Nice to see your addon to the if statement works :)
Tho, I still think that when we have micros() just before overflow and next_aq = an overflowed value, like 1, we get:
micros() > next_aq, so it acquires right there one acquisition, instead of waiting to micros()  to overflow and get to next_aq!

Quote
I would recommend you to test the behavior as it is especially important for your project.
I already had different projects, in which a problem only occurs irregularly and it is very difficult to analyze this later
What do you mean?

Go Up