I need to store approximately 250,000,000 values from a sensor (here potentiometer) and store them in a file on SD card.. i wrote this piece of code.. but it reads values only till some 3500.. if i increase it beyond that to even 4000, it gives this error..not enough memory .. I am using arduino MEGA 2560 as of now.. I have DUE board also in case MEGA wont suffice ..
I have also attached the code alongwith..
Basically do not store the data in the arduino memory there is not enough room even in a Due. Just write the data to the card as each sample comes in.
If that is not fast enough for you then an arduino is not the right board for your project.
I did not say without reading it through the arduino I said without storing it in the arduino.
So you get a sensor value and write it to the file, then you do that again and again until you have all your samples stored then you close the file.
yeah .. i get that but when I read it , it automatically gets stored in memory in an array in my code.. I m not specifically storing it.. How should I kind of skip that.. Here's my code for your reference.. I am reading values in an array a[] , and storing them simultaneously in a file called 'TEST'.. Is there any command for clearing it from the memory..or can you please tell me how to modify this code so that it just reads the values and writes it on SD card..
After 32000 values ..say at 35000 values..it goes into infinite loop.. no error.. nothing.. any ideas why it is probbly behaving like that..using arduino MEGA 2560..
What is the SD card type and size you are using, and What file system are you using on the SD card (fat16, fat32, etc) ?
Is it possible that you are trying to create a file size above the maximum allowable on the SD card?(There are limitations for fat16 of 4gb per file iirc, maybe closing the file and creating another one after it reaches a certain size is a possible solution.)
Is it possible that you are overflowing the onboard memory with data?
How often are you closing and re-opening the file you are creating on the SD card. ( closing the file will force the data to be written to the file system on the SD card, hopefully freeing up the ram on the board).
Are you allowing enough time for the data to be written to the SD card? (250,000,000 variables sounds like a lot of information and may take some time to accomplish on a mc)
Not sure if any of this is helpful, just brainstorming.
probably it makes sense to fill a buffer e.g. 64 bytes or 256 bytes (sector size of SD card) and write the buffer in one write.
That takes probably approx as long as a write of a single byte.
Printing the file as text using println() takes more bytes (3-7bytes) as when you write the data in binary format (2 bytes).
Now I need to store the values read into the text file back to arduino for processing..
I want to read values in an array of size = SRAM memory .. store their result in arduino and read the next set of values from the file..
My problem is values are stored n a text file line by line.. and I am not able to read them line by line..
I have written a code that can read values if they are stored one after the other .. and I can't think of a way to read them line by line..
I am attaching my code, the file it can read (TI.txt) and the one it cannot read (VMA.txt)
I am reading 20 values for testing (3 digits form 1 value )..
Is there any way I can modify this code to read values line by line..
I found a function called parseInt but I am not able to use it in my code..
Please help !!
As for buffering I tried to read values in double loop ..size of inner loop=63=assumed size of sector in SD CARD.. but inner loop reads only 1 value..I have attached the code alongside..
My problem is values are stored n a text file line by line.. and I am not able to read them line by line..
So, you read one character at a time from the file. If that character is not a carriage return or line feed, store it in the next position in an array.
If it is a carriage return or line feed, process the data in the array, reset the array, and discard the character.
Why do you want to store it is a text file just store the raw values. I am not the first to suggest this but you seem to be ignoring a very good helpful tip. Also doing your own buffering does help slightly as it reduces overhead by calling the function. If you look at the source code you will see there is a fair amount of code that is executed before appending to the buffer. What I would do is create a 1024 byte buffer then put the reading code in an interrupt. When the buffer gets half full (you have read 512 bytes) write the first part of the buffer the continue reading the sensor values to the second half. Once you have read 512 bytes more of data write the second half and while doing that read data to the first half of your buffer. This is called a circular buffer. So why would you go through all the trouble doing this you may ask. The reason is that this way you can ensure that the data was read at a regular known time interval. If you read a certain amount then write the values will be irregularly spaced time wise. Also don't use SD.h instead use sdfatlib see Google Code Archive - Long-term storage for Google Code Project Hosting. the reason I recomend sdfatlib over SD.h is because SD.h is basically a wrapper around an old version of sdfatlib that contain bugs that have since been fixed. Also when using SD.h you will wear your sd card very fast because every write causes a sync meaning that the sector is read then the data you want changed is changed and the sector is erased and rewritten. Sdfatlib can sync every write if you want it to but it is not forced on. Also when using the .print family of functions use the F() macro around constant text. This will save on ram.
Thanks for the cool idea.. it seems great but I am a beginner with coding and microcontrollers so have many queries..
By buffer , do you mean an array??
If yes then after reading 512 bytes.. say from 0-511.. it will write the 0th byte to the card.. and simultaneously read the 512th byte from sensor.. then write 1st byte to the card simultaneously reading 513th byte from sensor..
Doing this it is effectively writing one byte at a time only that was what my previous code was doing..
If not.. do you mean it will read 512 bytes.. then as it reads the next few bytes..it will simultaneously write the 512 bytes at once and not byte by byte ..then when the array again has 512 bytes it will again raise interrupt , then write the 512 values and and keep reading..
If this is the case I don't understand how will the 2 tasks..reading from sensor and writing to SD card be synced.. as in how many bytes will it read as the writing process takes place .. and how will the buffer write 512 values at a time.. because print and println commands write 1 value at a time.. isn't it?
If possible can you please help me with a code segment of this..
What do you mean by raw data .. where should I store data then..or write to SD card as what??
When I write data byte by byte.. instead of writing 1 byte it writes 3 bytes.. first 2 of which are junk..
means if I run a loop for reading 5 values and side by side printing them on SD card .. it reads 5 values prints 5 values on serial monitor.. but writes 15 values on SD card.. first 10 of which are junk and last 5 correspond to the values displayed on Serial monitor..
I checked it by printing the value I want to write on the serial monitor .. I have attached the code alongside..Has it go anything to do using SD.h and not sdfatlib.h.. ??
To answer your questions by buffer I do mean array. And yes you read the values one by one but write 512 bytes all at once. I could help you but first I want you to grasp the concept of raw data. Raw data is just storing the numbers without doing any conversion. You can write a PC program to do the conversion if you need to. To see the values you can open the file created with a hex editor.
I do not know why you are getting junk values. I do not think it is because of SD.h but regardless I rewrote your code to use sdfatlib and cleaned it up. Lets first worry about you being able to interpret the data. I made the loop run ten times instead of one as it is pointless to have a loop that runs once. You should expect to get a 20 byte file.
#include <SdFat.h>
//int cs_pin=10;
#define cs_pin 10
//int p_pin=A0;
#define p_ping A0
/*Normally I don't recommend the use of global variables however if you wish to
* use these in an interrupt or loop you cannot pass parameters so that is why they are global */
SdFat sd;
SdFile file;
// store error strings in flash to save RAM
#define error(s) sd.errorHalt_P(PSTR(s))
void setup(){
Serial.begin(115200);
Serial.println(F("Initializing Card"));
pinMode(p_pin,INPUT);
if(!sd.begin(cs_pin, SPI_FULL_SPEED))
sd.initErrorHalt();
if(!file.open("var.bin", O_WRITE))
error("Cannot open file");// Error will go into a loop
Serial.println(F("Card Ready"));
unsigned char z;//If you want to have a longer loop use a larger variable
for(z=0;z<10;++z){
unsigned r=analogRead(p_pin);
file.write(&r,sizeof(r)); // write two bytes. When using println they are converted to text this does no conversion
Serial.println(r);
}
file.close();// When completed it is very important to close the file
Serial.println(F("Done.. PARTY"));
}
void loop(){
}
The code looks fine to me. How are you connecting the sd card? If you are using resistor voltage dividers you may need to change SPI_FULL_SPEED to SPI_HALF_SPEED. The extension does not matter. I just picked .bin for binary.
