Reading 250,000,000 values from sensor and storing it on SD card

The code you just posted compiles with no errors on my Due.

Edit: it even runs after I change the file name to 8.3 format:

acquisition done
Card Ready
start time=
83133total aquisition time=
484628Done.. PARTY
acquisition done
Card Ready
start time=
83647total aquisition time=

The linker must remove the global z[24000].

The times make no sense because the array of 24,000 int uses too much memory. This program writes over some system memory.

It has a lot of other problems.

Here is a run with z[20000] and a few fixes:

acquisition done
Card Ready
start time=84873
total aquisition time=376344
Done.. PARTY
acquisition done
Card Ready
start time=553122
total aquisition time=400856
Done.. PARTY
acquisition done
Card Ready
start time=1045464
total aquisition time=399503
Done.. PARTY

It appears as though we were debating over something that should not even exist in his code. I have already suggested solutions that solves his problems and problems he did not know even existed. Which was to use a circular buffer and interrupt driven code to do the sampling. I'd imagine that he will be doing some kind of statistical analysis on these numbers and having each sample being read at a regular interval makes things much easier to do. When using a circular buffer you don't need such a large array. One thing I am wondering is what is he going to do with the 250000000 values? Where did the number 250000000 come from?

@ edison member.. thanks a lot.. the problem was the file name..
its working fine but

  1. It is overwriting values because only 24000 values are stored in the file not 24000*15=3600,000..
    I tried writing O_APPEND to continue writing values ..but same result..

  2. Also.. I am using a loop to write the read values stored in an array to write these values to the SD card.. Is there any command to directly write the values stored in the array to the card without using the loop i.e writing all value at a time instead of writing them one by one..

@mr arduino..

i need to sample flowing blood at a very high rate and for efficient results .. i ll need 1 million samples atleast..and i would need same 50,000 samples in one second continuously (ie sampling speed for storing continuously =50,000 corr to the 24000 i m using now) .. a delay of a few milliseconds (i.e i guess the order of time that would be taken to write these 50,000 samples to the SD CARD) after 50,000 samples won't matter in my project.. but its important that i get 50,000 samples in 1 second.. and say take 20 such sets ..(50000*20=1, 000,000 corr to 250,000,000 finally)

i left writing that circular buffer code in between delay of a few milliseconds after recording 50,000 is not important .. i m not experienced with coding.. though i would very much like to try both these algos and compare their pros and cons on my real time project.. and am trying them.. so just started to read examples in SdFat library.. :slight_smile:

i ll need 1 million samples atleast..and i would need same 50,000 samples in one second continuously

There is no point in logging a time series of one million samples in a program like yours. The information content of a time series depends as much on the time a point is logged as the value. There are mathematical treatments of the information content of such a time series but I won't go to that level. Just think of plotting a two dimensional curve in the x-y plane. It won't be correct unless you know both x and y for each point.

SD cards have random write delays of up to 250 milliseconds so you may have huge time uncertainty in your data.

Here is a program that triggers the ADC with a timer to eliminate time jitter, reads the ADC in an interrupt routine to avoid data loss, buffers the data for a background process that writes it to an SD Try this super fast analog pin logger - Storage - Arduino Forum.

The above logger is for an AVR Arduino. You need something like this for Due.

I tried writing O_APPEND to continue writing values ..but same result..

remove the O_TRUNC, it truncates the file to be empty.

if(!file.open("pleasework.txt", O_WRITE|O_CREAT|O_TRUNC))

is there any command to directly write the values stored in the array to the card without using the loop i.e writing all value at a time instead of writing them one by one..

Write the entire array in binary and convert the file to text later. Make your buffer a multiple of 512 bytes, the size of an SD block. The optimum size is 32KB, the cluster size on a properly formatted SDHC card.

int SdFile::write (const void* buf, size_t nbyte )
Write data to an open file.

Note
Data is moved to the cache but may not be written to the storage device until sync() is called.

Parameters
[in] buf Pointer to the location of the data to be written.
[in] nbyte Number of bytes to write.

Don't use an array of int since your data is under 16-bits, use a 16-bit short array. This will save memory and reduce the size of the binary file.

i ll need 1 million samples atleast..and i would need same 50,000 samples in one second continuously

You don't need a micro-controller, You need a computer.

@ fatlib16..

Thanks a ton for the so very helpful answers .. I haven't worked out all that you've told(will hopefully be able to try that in some time :P) .. just checked the code after removing
O_TRUNC..

It runs the outer loop (ch loop) twice instead of running it once.. and the first time values are junk ..(checked by printing values on serial monitor.. attached modified test code along !!)..

@steinie44..

I can take 48000 samples in a second through arduino DUE using type using short int..

tryingagain.ino (1.67 KB)

@steinie44..

I can take 48000 samples in a second through arduino DUE using type using short int..

Yes, but can you take a million samples in 20 seconds and store them at the same time?

I can take 48000 samples in a second through arduino DUE using type using short int..

Of course you are building a very complicated noise generator if you pursue your current program. You must study signal to noise issues for time sequences taken with an ADC. At 50,000 samples per second you need time jitter in the nanoseconds.

The formula for maximum time jitter is:

t = 1/(3.14f2^(n+1))

Where f is the sampling frequency and n is the number of bits of accuracy for the ADC.

For f = 50,000 and n = 10 bits:

t = 3.11 nanoseconds

Fast data acquisition is not about coding but rather understanding the theory.

You have no idea what things like system tick and SD random delays will do.

i ll need 1 million samples atleast..and i would need same 50,000 samples in one second continuously

You don't need a micro-controller, You need a computer.

This can be done with a modern ARM Cortex M chip but you need need good design and an RTOS would help. I am experimenting with a fast STM32 chip with a 2.4 MSPS ADC. I can trigger the ADC using a timer so jitter is less than one cycle for a 168 MHz CPU. the ADC can be used with a nice multi-buffered DMA controller and I can write the data to an SD easily in the background without interfering with the ADC.

ohh...Ok.. I ll check that signal and noise issues ..
but why is the outer loop running twice as many times and giving junk values?

ohh...Ok.. I ll check that signal and noise issues ..
but why is the outer loop running twice as many times and giving junk values?

Your program still had lots of issues like initializing the SD card in the loop. You should not be opening and closing the file the way you are.

You have now change loop indices to short int. short int can only range from –32,768 to 32,767. Who knows what this is doing Use size_t for things like loop indices.

Why are you pursuing this junk program? The current structure will never be satisfactory.

ok.. i ll read examples of analog data logger and SdFat library and then code :stuck_out_tongue:

ADC speed:
http://www.microsmart.co.za/technical/2014/03/01/advanced-arduino-adc/

ok.. i ll read examples of analog data logger and SdFat library and then code

The SdFat examples won't help the data logger example is only good for about one sample per second.

You want to take data 50,000 times faster. This is the difference between a cow walking around in a pasture and a Mars or Moon lander at max speed. I don't think your cow will jump the Moon. You can't do this with stock Arduino programming.

ADC speed:
http://www.microsmart.co.za/technical/2014/03/01/advanced-arduino-adc/

This article is for the AVR ADC so doesn't apply to Due. The Due ADC is over 20 times faster than the AVR ADC.

The problem is that the Arduino Due analogRead function will never be adequate for the time jitter you need. You need a lot more than reading a few hobby level Arduino programs.

Here is the ($24 from Future Electronics) STM32F429 board I am using http://www.st.com/web/catalog/tools/FM116/SC959/SS1532/PF259090.

It's almost trivial to solve your problem with this board if you are a skilled STM32 programmer.

The board has an 8MB RAM device that is accessible by the DMA controller. You just setup a timer, the ADC, and a DMA channel and acquiring up to 4 million samples is almost automatic. The board has three ADCs and each ADC can digitize samples at 2.4 million/sec.

The three ADCs can run in interleaved mode to digitize one channel at 7.2 million samples/sec.

Too bad you don't have the skills to use this board. It kills Due at 180 MHz, 2 MB flash, 256 KB fast RAM, and 8 MB SDRAM.

Yeah.. this MCU has cool features..much btr than arduino DUE.. but I dn have that long to complete this assignment..

The thing that I am unable to understand is .. what is the problem with arduino due.. I can use unsigned short int and store 48000 values in one second easily.. the only problem is due to some reason the code repeats twice or thrice..
tough I don't see any glitch in the logical part of the program.. You said my code was junk !! :stuck_out_tongue: Please tell me how can I improve it..

So if I make my code better.. arduino should be able to solve the purpose.. Isn't it..

I ve also got Raspberry pi if it is easier on that..though i ve never used it till now .. !!

I ve also got Raspberry pi if it is easier on that..though i ve never used it till now .. !!

The Pi does not include a hardware analog to digital converter, but a external ADC (such as the MCP3008) can be used along with some bit banged SPI code in python to read external analog devies. Here is a short list of some analog inputs that could be used with this setup.

Raspberry Pi won't come close for your needs.

You know v_m_a everyone started out as a beginner at some point in their lives.
However (in part) the way to take yourself from beginner to more advanced is to actually listen to what people have to say. I have seen some instances where people have given some suboptimal advice on this forum but that is rare and if that is the case someone else can correct that. The issue here is you have a what appears to be in your opinion an important project that sounds like it is for the school that you go to. If I had an important project with a deadline and I had your current lack of programming ability I would try to avoid relying on other people to do all the work for you. I would only post a question as a last resort. I would do my own research and improve my own skills. How I learned to code was I had a goal in mind and I told myself I want to do this. So what did I do? I used my favorite search engine and found out that there were tutorials that could teach me how to do what I want. For example if I want to read a file on the PC and I am using c I could search for "C file tutorial" (without the quotes) and this was the first result. C File I/O Tutorial - Cprogramming.com I am already familiar with this stuff I just used this as a simple example. It took me around ten seconds to find that by the way. I do hope you continue to program (assuming continue implies improving and learning) and I am sure a few years down the road you will check your post history and think something to the effect of "wow what was I thinking?". I admit I can say that I have had that experience. So when fat16lib tells you that your program is junk (which may be holding back) and you ask how you can improve that is good to hear but you need to be also asking yourself that not just other people expecting them to do everything for you also keep in mind that multiple members have given you helpful suggestions that you have ignored. You know I have not read the datasheet for the SAM3X8E much as I do not know own anything that contains one or have any plans for it but I think an interrupt driven code using dma is your best bet. You should be the one reading the datasheet not I. I read the datasheet for my projects.