Reading 250,000,000 values from sensor and storing it on SD card

Thanks a ton guys..

Changing int to long worked !!

to the brainstorming guy.. :stuck_out_tongue:
I am using fat32.. n i had checked the rest of the stuff.. thanks..

n do u knw how to store it in a buffer and write..it will increase the speed a lot..which is kind of one of the most crucial issues wth my project..

n do u knw how to store it in a buffer and write..it will increase the speed a lot..which is kind of one of the most crucial issues wth my project..

Writing to the SD card is already buffered. Adding your own buffering won't help.

Now I need to store the values read into the text file back to arduino for processing..
I want to read values in an array of size = SRAM memory .. store their result in arduino and read the next set of values from the file..

My problem is values are stored n a text file line by line.. and I am not able to read them line by line..
I have written a code that can read values if they are stored one after the other .. and I can't think of a way to read them line by line..
I am attaching my code, the file it can read (TI.txt) and the one it cannot read (VMA.txt)

I am reading 20 values for testing (3 digits form 1 value )..
Is there any way I can modify this code to read values line by line..
I found a function called parseInt but I am not able to use it in my code..
Please help !!

VMA.TXT (9.77 KB)

T1.TXT (66 Bytes)

working_code.ino (1.07 KB)

As for buffering I tried to read values in double loop ..size of inner loop=63=assumed size of sector in SD CARD.. but inner loop reads only 1 value..I have attached the code alongside..

double_loop.ino (748 Bytes)

My problem is values are stored n a text file line by line.. and I am not able to read them line by line..

So, you read one character at a time from the file. If that character is not a carriage return or line feed, store it in the next position in an array.

If it is a carriage return or line feed, process the data in the array, reset the array, and discard the character.

Why do you want to store it is a text file just store the raw values. I am not the first to suggest this but you seem to be ignoring a very good helpful tip. Also doing your own buffering does help slightly as it reduces overhead by calling the function. If you look at the source code you will see there is a fair amount of code that is executed before appending to the buffer. What I would do is create a 1024 byte buffer then put the reading code in an interrupt. When the buffer gets half full (you have read 512 bytes) write the first part of the buffer the continue reading the sensor values to the second half. Once you have read 512 bytes more of data write the second half and while doing that read data to the first half of your buffer. This is called a circular buffer. So why would you go through all the trouble doing this you may ask. The reason is that this way you can ensure that the data was read at a regular known time interval. If you read a certain amount then write the values will be irregularly spaced time wise. Also don't use SD.h instead use sdfatlib see Google Code Archive - Long-term storage for Google Code Project Hosting. the reason I recomend sdfatlib over SD.h is because SD.h is basically a wrapper around an old version of sdfatlib that contain bugs that have since been fixed. Also when using SD.h you will wear your sd card very fast because every write causes a sync meaning that the sector is read then the data you want changed is changed and the sector is erased and rewritten. Sdfatlib can sync every write if you want it to but it is not forced on. Also when using the .print family of functions use the F() macro around constant text. This will save on ram.

Thanks for the cool idea.. it seems great but I am a beginner with coding and microcontrollers so have many queries.. :stuck_out_tongue:

  1. By buffer , do you mean an array??

If yes then after reading 512 bytes.. say from 0-511.. it will write the 0th byte to the card.. and simultaneously read the 512th byte from sensor.. then write 1st byte to the card simultaneously reading 513th byte from sensor..
Doing this it is effectively writing one byte at a time only that was what my previous code was doing..

If not.. do you mean it will read 512 bytes.. then as it reads the next few bytes..it will simultaneously write the 512 bytes at once and not byte by byte ..then when the array again has 512 bytes it will again raise interrupt , then write the 512 values and and keep reading..
If this is the case I don't understand how will the 2 tasks..reading from sensor and writing to SD card be synced.. as in how many bytes will it read as the writing process takes place .. and how will the buffer write 512 values at a time.. because print and println commands write 1 value at a time.. isn't it?

If possible can you please help me with a code segment of this..

  1. What do you mean by raw data .. where should I store data then..or write to SD card as what??

  2. When I write data byte by byte.. instead of writing 1 byte it writes 3 bytes.. first 2 of which are junk..
    means if I run a loop for reading 5 values and side by side printing them on SD card .. it reads 5 values prints 5 values on serial monitor.. but writes 15 values on SD card.. first 10 of which are junk and last 5 correspond to the values displayed on Serial monitor..

I checked it by printing the value I want to write on the serial monitor .. I have attached the code alongside..Has it go anything to do using SD.h and not sdfatlib.h.. ??

codecheck.ino (636 Bytes)

To answer your questions by buffer I do mean array. And yes you read the values one by one but write 512 bytes all at once. I could help you but first I want you to grasp the concept of raw data. Raw data is just storing the numbers without doing any conversion. You can write a PC program to do the conversion if you need to. To see the values you can open the file created with a hex editor.
I do not know why you are getting junk values. I do not think it is because of SD.h but regardless I rewrote your code to use sdfatlib and cleaned it up. Lets first worry about you being able to interpret the data. I made the loop run ten times instead of one as it is pointless to have a loop that runs once. You should expect to get a 20 byte file.

#include <SdFat.h>
//int cs_pin=10;
#define cs_pin 10
//int p_pin=A0;
#define p_ping A0
/*Normally I don't recommend the use of global variables however if you wish to
 * use these in an interrupt or loop you cannot pass parameters so that is why they are global */
SdFat sd;
SdFile file;
// store error strings in flash to save RAM
#define error(s) sd.errorHalt_P(PSTR(s))
void setup(){ 
	Serial.begin(115200);
	Serial.println(F("Initializing Card"));
	pinMode(p_pin,INPUT);
	if(!sd.begin(cs_pin, SPI_FULL_SPEED))
		sd.initErrorHalt();
	if(!file.open("var.bin", O_WRITE))
		error("Cannot open file");// Error will go into a loop
	Serial.println(F("Card Ready"));
	unsigned char z;//If you want to have a longer loop use a larger variable
	for(z=0;z<10;++z){
		unsigned r=analogRead(p_pin);
		file.write(&r,sizeof(r)); // write two bytes. When using println they are converted to text this does no conversion
		Serial.println(r);
	}
	file.close();// When completed it is very important to close the file
	Serial.println(F("Done.. PARTY"));
}

void loop(){
	 
}

Thanks a lot for helping me with the code..

It is getting compiled but at runtime it shows error "cannot open file"..

Also I didn't get where will it will store values.. is .bin a kind of file that will be created?

The code looks fine to me. How are you connecting the sd card? If you are using resistor voltage dividers you may need to change SPI_FULL_SPEED to SPI_HALF_SPEED. The extension does not matter. I just picked .bin for binary.

I am just directly connecting SD card to arduino DUE using SD card shield .. I have not externally connected any resistor or anything else.. though I dont know if they are internally connected on those pins.. In any case it not able to open the file at half speed command also..

and the code is fine cz its getting compiled and all.. but error comes while opening the file..
and in .bin file the data would be saved in zeroes and ones form?

If you using a shield or short wires it should work at full speed. Try running benchSD.ino and tell me what it does. While looking at that sketch I noticed an extra

delay(400);  // catch Due reset problem

Before the line involving sd.begin try adding that delay to the code I posted above.

benchSD.ino file runs fine i guess.. i ve attached the output file along in.docx form..it was a .dat file that ws generated.. i gave input as 3..
n after adding delay of 400ms also..the cdode is showing the same error at runtime.. "cannot open file.. "

output of benchdat.docx (484 KB)

Sorry I told you to run the wrong sketech. Try running bench.ino. benchSD uses the faulty SD.h. Also it may be failing because the file already exists try changing (in the code that I posted) from O_WRITE to O_WRITE|O_CREAT|O_TRUNC
Tell me if that works.

thanku a million times.. :slight_smile: :stuck_out_tongue:

its working after changing .. O_WRITE to O_WRITE|O_CREAT|O_TRUNC .. I tried to write it to bin file first .. but i wasn't able to read that bin file.. i ve attached it along.. so i changed filetype from bin to txt and changed "file.write(&r,sizeof(r)) to file.println(r) and file.print(r) for printing consecutively ..

And I dunno how :stuck_out_tongue: but its working perfectly .. It is storing the same 10 values in the text file as it is printing on serial monitor..no junk values.. :smiley: THANKU..
just dont know how it worked and why mine wasn't working..

thanks a lot!!.. so level 1 done.. :stuck_out_tongue:
now please help me with the code of storing in a buffer .. and writing 512 values at a time too.. :stuck_out_tongue: ..

n I ll try to understand this code perfectly..and see if I can figure out why mine wasn't working and taking junk values..
and that bench.ino is also working i guess.. i ve attached all the outputs in case u wanna have a look

MRAROCKS.TXT (30 Bytes)

MRARDUINO.TXT (50 Bytes)

VAR.BIN (40 Bytes)

bench.docx (488 KB)

If you must use a text file at-least separate each reading with a comma. That is yet another disadvantage of a text file. You need a separator because 1000 is different length than 999. When saving the number without converting to text each reading is a constant two bytes (in this instance). I have attached a screen shot illustrating raw binary data. Maybe this will help you understand. Binary file - Wikipedia

Try this:

http://forum.arduino.cc/index.php?topic=228549.0

ok.. Got it about binary files..
as for 999 and 1000.. all my input values would be in the range of 500-800.. so all are 3 digit only..
so that won't be an issue..
I don't have problem with any file type as long as it stores data reliably and can read it back to process it..

and now I want to READ ABOUT 250,000,000 values and store it in on SD card..
I had written that code with SD.h and it was taking in junk values..but SdFat.h doesn't ..
I read the thread you had provided link to and started reading the library too.. but its too long and I don't think that would be the correct way to write the code..

Could you please write the code for reading and storing values using this library commands..
I know I should try it first myself but it seems like it ll take a whole week if I read through the library and examples and try to understand it !! and I am sort of running short on time from my project..

That is just sheer laziness. I do not want to do everything for you. All you need to do is setup an interupt that reads the ADC and fills a buffer. Then in the main loop wait until the buffer is half full and do a write. Also there is a difference that should and will. What if you get something other than three digits. Also why do you not like a binary file. I do not know of any program that will accept the data as is. If you add a comma many programs that support CSV can open it. Either way you may end up writing a converter so at this point you might as well use a binary file. Also yes that thread does have some good code just modify it for your needs.

You are right..
I WILL try first now.. z I SHOULD.. :slight_smile: