Go Down

Topic: Unformatted write to SD (Read 7799 times) previous topic - next topic

ardnut

hmm, perhaps sleeping longing enough for SPI unit to finish transfering the last byte it was sent before triggering ADC would avoid breaking the transfer?

pito

#16
Aug 18, 2012, 11:44 am Last Edit: Aug 18, 2012, 07:11 pm by pito Reason: 1

Quote
Doesn't that limit you to 1kS/s unless you are willing to accept some substantial jitter?

The jitter for the 100,000 sample per second logger is a less than one CPU cycle, which is 62.5 ns.

FYI - with 62.5ns jitter and 50kHz input frequency your SNR cannot be better than 40.16db 34dB.

ardnut

#17
Aug 18, 2012, 12:46 pm Last Edit: Aug 18, 2012, 01:43 pm by ardnut Reason: 1
40dB? He's only sampling 8bit (or 10b in conditions that will only give 8b accuracy). Though interleaving ADC and SD writing at this speed is pretty good going for an Arduino. Kudos.

I need 10b accuracy and am looking for speed in order to over sample and do some filtering since a 10b sample is never 10b accurate.

I could reduce random noise by averaging but if I have some cyclic interference to filter out jitter becomes a worry.

The ENOB and other statistical evaluations depend upon sampling a smoothly varying signal on one channel. This will not reflect the accuracy of measurments done by muxing adjacent channels where successive values could potentially change from one extreme to the other.

How do you calculate the s/n effect of jitter? Do you have a reference ?

fat16lib

Quote
Specifically what happens when you go beyond the "management block" size of 128MB or whatever? Do you know how/why you were able to avoid hitting a busy delay?

The busy delay has nothing to do with "management block".  The big delay happens in single block mode because the controller does not plan ahead. For the 12th time I use multi-block streaming mode.
Quote
Yes, but have you tested >5MB to see whether you are still getting no busy time? 

Yes, I designed SdFat for audio recording and other high speed logging.  I have logged for hours in real apps.

Limor Fried asked me to make a version for beginning users.  The Arduino company decided it was too complex and wrapped it with their SD.h API.  That's why you think I designed SdFat for beginning user. 

You can used just the files Sd2Card.h, Sd2Card.cpp and SdInfo.h as a library.  This sketch takes 2120 bytes of flash and will write block zero of an SD.
Code: [Select]

#include <Sd2Card.h>
Sd2Card card;
uint8_t buf[512];
void setup() {
  card.init();
  card.writeBlock(0, buf);
}
void loop() {


Quote

ADC noise reduction mode which is required to get (nominal) 10b accuracy from the Atmel chip.  This scheme seems fine for your 8bit sampling but  you have to chose between higher resolution ADC and jitter.

Forget the datasheet, it is a general guide.  Look at the AVR evaluation tests. In the papers that I pointed to, the ADC is triggered by the CPU clock and noise reduction is done using the DIDR.

The result is 7.4 ENOB for the 2MHz rate used at 100,000 samples per second.  For 10-bit sampling 33 ksps gives a ENOB of about 9.3 with a 500kHz ADC clock.   The max ENOB is 9.5 for the AVR ADC in any test.

Here is a paper on SNR due to sampling jitter http://www.analog.com/static/imported-files/tutorials/MT-200.pdf.

Clearly the jitter in the AVR timer compare event is less that a CPU cycle.  I said it was less than a CPU cycle since I don't know the exact number.

ardnut,  If you are over sampling, why write all the data to the SD?  What kind of signal are you recording?  How can you possibly use the AVR ADC for a fast multi-channel signal?

ardnut

#19
Aug 18, 2012, 05:36 pm Last Edit: Aug 18, 2012, 05:55 pm by ardnut Reason: 1
Quote
Forget the datasheet, it is a general guide.  Look at the AVR evaluation tests. In the papers that I pointed to, the ADC is triggered by the CPU clock and noise reduction is done using the DIDR.


As an engineer "forget the datasheet" is not part of my way of working. It is not a general guide , it's the bible. Evaluation tests like the audio site your cribbed the graph from are valuable sources of information. They do not negate the datasheet. In fact they do not even contradict it since they are not giving the same information.

I get the impression you are confusing a statistical measurement , ENOB, with the spec for the accuracy on one ADC conversion. The two results are compatible and not contradictory. They are different things.

I am logging physical quantities, not audio. You probably have different criteria for what you are doing. For you, jitter is probably more important than ADC accuracy. In my case it's somewhat the opposite. I require high absolute accuracy; the exact microsecond timing of each datum is less important (other than the question of possibly filtering cyclic interference I mentioned earlier which is more likely 50Hz than 50kHz).

I do not need the root sum error over a few thousand readings , I need to know the accuracy of one conversion. ENOB , S/N etc are useful supplements that help in looking at the  S/N of the real signal vs the S/N of the ADC and what measures I need to take to get within spec.
Quote

and noise reduction is done using the DIDR.

DIDR is just one measure to reduce internal noise. That is no way the same thing as using ADC noise reduction mode, both are required to attain get the specified accuracy. (See the spec sheet for detains ;) )

Are you able to comment on whether sleeping the CPU would break the SPI streaming or is the protocol robust enough to stand  a circa 100us hiatus?



pito

#20
Aug 18, 2012, 07:13 pm Last Edit: Aug 18, 2012, 07:16 pm by pito Reason: 1
>How do you calculate the s/n effect of jitter? Do you have a reference ?

SNR = 20 * log ( 1 / (2 * PI * f_signal * t_jitter ) )

reference - see above analog dev appnote, I've seen also the calculus without "2" - therefore 40db or 34db :)

fat16lib

Quote
As an engineer "forget the datasheet" is not part of my way of working. It is not a general guide , it's the bible.

As a PhD physics researcher working on some the world's largest experiments, I take datasheets with a grain of salt.  They are a clue but you must evaluate devices in the mode you use them.  Often devices change but not the datasheets.
Quote

I get the impression you are confusing a statistical measurement , ENOB, with the spec for the accuracy on one ADC conversion. The two results are compatible and not contradictory. They are different things.

Accuracy of one conversion is a statistical measurement.  Accuracy is always a statistical measurement.

Did you read the Oskar Leuthold paper on the AVR ADC?  He is an engineer working for GEC Plessey Semiconductors.  He designs fast, 200 Megasample per second  A/D converters.

What is wrong with his testing of the AVR ADC?
Quote

I am logging physical quantities, not audio.

Audio is a physical signal.  Some of the most advanced research in high quality ADCs involves audio.
Quote

Are you able to comment on whether sleeping the CPU would break the SPI streaming or is the protocol robust enough to stand  a circa 100us hiatus?

Using ADC Noise Reduction Mode is likely to cause problems with an SD transfer.  Atmel documents suggest that Idle Mode could work O.K.  The SPI controller is stopped in ADC Noise Reduction Mode and that could cause a glitch.

That's a comment not a proven fact.  My original library allowed interruption of block transfers but I kept having problems so I removed the code.

If you need accuracy at 50 Hz why not us an external ADC?  I use external sigma-delta ADCs frequently.  I wrote a fast Software SPI library that runs at about 2MHz to access external ADCs since it is difficult/impossible to share the hardware SPI bus with the SD.  

At 50 Hz you could even use low cost I2C ADCs.  Here is a family of 18-bit delta-sigma ADCs I like a lot (maybe too slow for you). They have a 0.05% on-board voltage reference and an on-board PGA. http://ww1.microchip.com/downloads/en/devicedoc/22088c.pdf.  It could give you 14 bits at 60 sps.

This may not work for your application but there are many choices that are easy to use.  Why fight with the AVR ADC if it doesn't meet your requirement.

I mainly play with pushing the AVR ADC as a game.  For serious hobby measurements I use external ADCs/sensors.  

I am now mostly using Cortex M4 STM32 processors, not Arduino.  I run ChibiOS/RT as the OS.

Here is an example board https://www.olimex.com/dev/stm32-h407.html it has three really fast 12-bit ADCs that can run in parallel.

pito

>Did you read the Oskar Leuthold paper on the AVR ADC?
A comment only - for a proper measurement with FFT you need:
1. a good antialiasing filter (first order RC is not enough)
2. to apply a "windowing" function before FFT (not done)
3. to average the spectra as the spectra accuracy relates to sqrt(N) where N is the number of FFTs
4. noise free voltage reference...

ardnut

Thanks for all that excellent information and the info on better external ADCs.

I'm just putting together a test project to see what Arduino/Atmel is capable of doing. I'm not just out to make life hard for myself, rather to see what it's limits are and see how much use it could be to me.

If your experience was that SPI cannot be made reliable in noise reduction mode that's going to seriously limit the logging capabilities at 10b. But that's what I'm out to establish.  Part of the appeal of Atmel was the onboard 10b ADC. If I have to add external ADC chips I may be asking "why Atmel"?


One quick aside: why log FAT access time in SdFatlib?  I usually mount such devices with the noatime option on Linux. Reading the device makes another hit on the wear cycle and few people are interested in last read access times (especially in this context). Maybe a configure option?


fat16lib

I don't log access time for read.  I provide a function that allows all time fields of a file to be set with a call like Linux/UNIX touch.

I have never been asked to provide an update to access time for reads.

For write/create I update all fields since it's free.  The directory entry only gets updated when you call sync() or close a file.  That's when time fields get updated in the directory entry.

I think the answer to why Atmel is Arduino.  Arduino give easy software/hardware access to microprocessors.

AVR is old but fun.  It reminds me of my first microprocessor hardware projects with the 6800 and 6502 (the 6502 was used in the Apple II) around 1974.

People use sleep modes with SdFat to save battery power.  They just aren't sleeping in the middle of a block transfer.  I think sleeping between blocks in multi-block mode would work.  The SD card won't know and chip select is high.

ardnut

Quote
Using ADC Noise Reduction Mode is likely to cause problems with an SD transfer.  Atmel documents suggest that Idle Mode could work O.K.  The SPI controller is stopped in ADC Noise Reduction Mode and that could cause a glitch.

That's a comment not a proven fact.  My original library allowed interruption of block transfers but I kept having problems so I removed the code.


ADC Noise Reduction Mode stops the IO clock. If there is a byte transfer under way this may well mess up synchronisation and cause the SD to lose the byte.

Do you think the following strategy would work:

When the timer trips to start an ADC conversion, first test SPIF:

Code: [Select]
if (!SPIF) {
  set SPI_STC
}
else {
  set ADC interrupt
  start ADC in NR mode
}
 
spi_stc_ISR()
{
  set ADC interrupt
  start ADC in NR mode
}


CS will remain low during circa 100us hiatus with no activity on SCK but no byte transfers will get interrupted.

To my limited  understanding of the subtleties of all this, that would seem to ensure reliable SPI transmission and keep the SD slave awaiting the next byte of the block.

NR mode will also stop the timer, but for a fixed known delay, so this should not add to jitter. It may be desirable to adjust the timer by 13.5 ADC cycles and look at the impact on anything relying on millis().


fat16lib

#26
Aug 19, 2012, 05:21 pm Last Edit: Aug 19, 2012, 06:01 pm by fat16lib Reason: 1
I did a test and the good news is reading the ADC in Noise Reduction Mode in the middle of an SD block transfer seems to work OK.

I did not write rawAnalogReadWithSleep() but it seems to use Noise Reduction Mode.

I added the code between the slashes to the write loop (two bytes are sent for each pass to slightly speed the transfer):
Code: [Select]

 for (uint16_t i = 0; i < 512; i += 2) {
   while (!(SPSR & (1 << SPIF)));
   //////////////////////////////////////////////////////////////
   int rawAnalogReadWithSleep();
   if (i == 300) {
     rawAnalogReadWithSleep();
     Serial.print('.');
   }
   /////////////////////////////////////////////////////////////
   SPDR = buf[i];
   while (!(SPSR & (1 << SPIF)));
   SPDR = buf[i + 1];
 }

I ran this sketch:
Code: [Select]

#include <SdFat.h>
#include <avr/sleep.h>
SdFat sd;
SdFile file;

ISR(ADC_vect) { }

int rawAnalogReadWithSleep() {
 // Generate an interrupt when the conversion is finished
 ADCSRA |= _BV(ADIE);

 // Enable Noise Reduction Sleep Mode
 set_sleep_mode(SLEEP_MODE_ADC);
 sleep_enable();

 // Any interrupt will wake the processor including the millis interrupt so we have to...
 // Loop until the conversion is finished
 do
 {
   // The following line of code is only important on the second pass.  For the first pass it has no effect.
   // Ensure interrupts are enabled before sleeping
   sei();
   // Sleep (MUST be called immediately after sei)
   sleep_cpu();
   // Checking the conversion status has to be done with interrupts disabled to avoid a race condition
   // Disable interrupts so the while below is performed without interruption
   cli();
 }
 // Conversion finished?  If not, loop.
 while( ( (ADCSRA & (1<<ADSC)) != 0 ) );

 // No more sleeping
 sleep_disable();
 // Enable interrupts
 sei();

 // The Arduino core does not expect an interrupt when a conversion completes so turn interrupts off
 ADCSRA &= ~ _BV( ADIE );

 // Return the conversion result
 return( ADC );
}

void setup() {
 Serial.begin(9600);
   // setup ADC
 analogRead(0);
 if (!sd.begin())sd.initErrorHalt();
 if(!file.open("ADC_TEST.TXT", O_RDWR | O_CREAT | O_AT_END)) {
   sd.errorHalt("opening test.txt for write failed");
 }
 for (uint16_t i = 0; i < 50000; i++) {
   file.println(i);
   if (file.writeError) sd.errorHalt("print");
 }
 file.close();
 Serial.println("Done");
}
void loop() {}

Lots of dots get printed and the file has the correct content.  The file has 660 blocks.

The bad news is that I can't find proof on the web that Noise Reduction Mode helps.

I tried various tests comparing analogRead() with rawAnalogReadWithSleep() above.  If there is an improvement it is really small.  Other factors overwhelm the the difference between the two functions.  Clean power to the Arduino makes a huge difference.

One curious result is that the two functions return slightly different results.  You need to average a 1000 measurements to see the difference.

My test setup was very crude so it is not definitive but as until I see proof I won't believe Noise Reduction Mode is worth the pain.

I hope you prove Noise Reduction Mode gives more accuracy.

Edit:
I did some more work and oversampling and Noise Reduction Mode are extremely frustrating.  If I work hard to reduce noise, oversampling will not work since you need noise for oversampling.  I don't trust oversampling with the 10-bit AVR ADC, it's too easy to fool yourself.

If you need more accuracy, an external ADC seem like a far better approach.


ardnut

#27
Aug 19, 2012, 06:51 pm Last Edit: Aug 19, 2012, 08:05 pm by ardnut Reason: 1
Thanks for testing that function. There's a couple of things that appear sub-optimal


was the second adc a typo? wouldn't this double read add jitter?
Quote

   int rawAnalogReadWithSleep();
   if (i == 300) {
//?      rawAnalogReadWithSleep();
     Serial.print('.');
   }


Code: [Select]

do
{  
 sei();
 sleep_cpu();
 cli();
}
// Conversion finished?  If not, loop.
 while( ( (ADCSRA & (1<<ADSC)) != 0 ) );


Now I'm confused. the interrupt triggers and the ISR only gets called when the conversion IS complete. What is he waiting for here, another not quiet ADC conversion ?


fat16lib

What appears sub-optimal?  I just stuck the lines in the loop to test sleep, not as a test of jitter.

I didn't post the Noise Reduction tests.

Appears your C++ is a bit rusty. The first statement is a type declaration so the library will compile.  I could have put it anywhere before the call.
Code: [Select]

  int rawAnalogReadWithSleep();

This is the call:
Code: [Select]

  rawAnalogReadWithSleep();


The "no-op" ISR is necessary to field the wake-up interrupt.

Most of the time in the loop is spent sleeping.  The person that wrote this function is allowing for wake-up by interrupts other than the ADC.  If the ADC is not done, the function goes back into ADC Noise Reduction Mode.

"no-op" ISRs are not uncommon.  Sometimes they clear a flag or cause other status change. They are very fast since no context needs to be saved.  I use one to clear a timer flag in the 100,000 sample per second logger.

I did more testing on the ADC Noise Reduction Mode.  I used a high resolution DAC to generate a ramp.  The DAC is on a well designed shield on the Arduino I was testing.

I got the noise so low with just analogRead() that I couldn't do oversampling.  For a number of DAC steps the 10-bit Arduino ADC always gives the same value.  I don't need noise reduction, I need noise injection to make oversampling work.

ardnut

#29
Aug 19, 2012, 09:21 pm Last Edit: Aug 19, 2012, 10:45 pm by ardnut Reason: 1
Quote
Appears your C++ is a bit rusty. The first statement is a type declaration so the library will compile.  I could have put it anywhere before the call.


Of course. More like my eyes so tired I can't even see straight after all this reading up and pouring through all this arduino pseudo code and pages of  libs full of #ifs and #buts and _D_OXYGEN__ garbage just to boil done to one-liner assembler code. It's enough to make my head spin.

Sorry for having thought you could have made a mistake.  ;)

Quote
The "no-op" ISR is necessary to field the wake-up interrupt.


Yes, I appreciate that. The arduino mumbo-jumbo ends up, after 2 or 3 pages of conditionals,  by converting the empty declaration into an inline  reti;

Quote
The person that wrote this function is allowing for wake-up by interrupts other than the ADC.  If the ADC is not done, the function goes back into ADC Noise Reduction Mode.


Ah OK. I was not thinking outside the simple case we were looking at. Indeed another ISR could have just been run , leaving the cpu active and ADC incomplete.

Makes sense.

Go Up