Go Down

Topic: Reading 250,000,000 values from sensor and storing it on SD card (Read 23999 times) previous topic - next topic

v_m_a

I need to store approximately 250,000,000 values from a sensor (here potentiometer) and store them in a file on SD card.. i wrote this piece of code.. but it reads values only till some 3500.. if i increase it beyond that to even 4000, it gives this error..not enough memory ..   I am using arduino MEGA 2560 as of now.. I have DUE board also in case MEGA wont suffice ..
I have also attached the code alongwith..

Code: [Select]

#include<SD.h>
#include<SPI.h>

int p_Pin=0;
int a[3999];
int CS_pin=53;
int count=0;
void setup()
{
  Serial.begin(9600);
  Serial.println("Initialising Card");
  pinMode(CS_pin,OUTPUT);
  pinMode(p_Pin,INPUT);
 
  if(!SD.begin(CS_pin))
  {
    Serial.println("Card Failed");
    return;
  }
  Serial.println("Card Ready");
  File dataFile=SD.open("test12.txt",FILE_WRITE);
  if(dataFile)
  {

for(int i=0; i<3998; i++)
{
   a[i]=analogRead(p_Pin);
   dataFile.println(a[i]);
   Serial.println(a[i]);
   count++;
    }
    Serial.println(count);
   }
    dataFile.close();
 
}

void loop()
{
  }


How can I overcome this memory error??  I  dont need to store these 250,000,000 values in arduino memory, just in a file on SD card..
Plzz help !!

Grumpy_Mike

Basically do not store the data in the arduino memory there is not enough room even in a Due. Just write the data to the card as each sample comes in.
If that is not fast enough for you then an arduino is not the right board for your project.

v_m_a

How should I directly store it without reading it through arduino ??

Grumpy_Mike

I did not say without reading it through the arduino I said without storing it in the arduino.
So you get a sensor value and write it to the file, then you do that again and again until you have all your samples stored then you close the file.

v_m_a

yeah .. i get that but when I read it , it automatically gets stored in memory in an array in my code.. I m not specifically storing it.. How should I kind of skip that.. Here's my code for your reference..  I am reading values in an array a[] , and storing them simultaneously in a file called 'TEST'.. Is there any command for clearing it from the memory..or can you please tell me how to modify this code so that it just reads the values and writes it on SD card..

Code: [Select]


#include<SD.h>
#include<SPI.h>

int p_Pin=0;
int a[3999];
int CS_pin=53;
int count=0;
void setup()
{
  Serial.begin(9600);
  Serial.println("Initialising Card");
  pinMode(CS_pin,OUTPUT);
  pinMode(p_Pin,INPUT);
 
  if(!SD.begin(CS_pin))
  {
    Serial.println("Card Failed");
    return;
  }
  Serial.println("Card Ready");
  File dataFile=SD.open("test.txt",FILE_WRITE);
  if(dataFile)
  {

for(int i=0; i<3998; i++)
{
   a[i]=analogRead(p_Pin);
   dataFile.println(a[i]);
   Serial.println(a[i]);
   count++;
    }
    Serial.println(count);
   }
    dataFile.close();
 
}

void loop()
{
 
}



v_m_a

kk.. finally got it..cn store it in a variable instead of an array..  :p

v_m_a

NEW TROUBLE!!

After 32000 values ..say at 35000 values..it goes into infinite loop.. no error.. nothing.. any ideas why it is probbly behaving like that..using arduino MEGA 2560..

cattledog

you need to declare i as an unsigned long rather than as an int.  You should be able to read  4,294,967,295 (2^32 - 1) values instead of 35,000.

Lander1979

#8
Jun 24, 2014, 09:34 pm Last Edit: Jun 24, 2014, 10:15 pm by Lander1979 Reason: 1
What is the SD card type and size you are using, and What file system are you using on the SD card (fat16, fat32, etc) ?

Is it possible that you are trying to create a file size above the maximum allowable on the SD card?(There are limitations for fat16 of 4gb per file iirc, maybe closing the file and creating another one after it reaches a certain size is a possible solution.)

Is it possible that you are overflowing the onboard memory with data?

How often are you closing and re-opening the file you are creating on the SD card. ( closing the file will force the data to be written to the file system on the SD card, hopefully freeing up the ram on the board).

Are you allowing enough time for the data to be written to the SD card? (250,000,000 variables sounds like a lot of information and may take some time to accomplish on a mc)

Not sure if any of this is helpful, just brainstorming.

robtillaart

some thoughts,

probably it makes sense to fill a buffer e.g. 64 bytes or 256 bytes (sector size of SD card) and write the buffer in one write.
That takes probably approx as long as a write of a single byte.

Printing the file as text using println() takes more bytes (3-7bytes)  as when you write the data in binary format (2 bytes).




Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

v_m_a

Thanks a ton guys..

Changing int to long worked !!

to the brainstorming guy.. :p
I am using fat32.. n i had checked the rest of the stuff.. thanks..

n do u knw how to store it in a buffer and write..it will increase the speed a lot..which is kind of one of the most crucial issues wth my project.. 

PaulS

Quote
n do u knw how to store it in a buffer and write..it will increase the speed a lot..which is kind of one of the most crucial issues wth my project.. 

Writing to the SD card is already buffered. Adding your own buffering won't help.
The art of getting good answers lies in asking good questions.

v_m_a

#12
Jun 25, 2014, 01:08 pm Last Edit: Jun 25, 2014, 01:18 pm by v_m_a Reason: 1
Now I need to store the values read into the text file back to arduino for processing..
I want to read values in an array of size = SRAM memory .. store their result in arduino and read the next set of values from the file..

My problem is values are stored n a text file line by line.. and I am not able to read them line by line..
I have written a code that can read values if they are stored one after the other .. and I can't think of a way to read them line by line..
I am attaching my code, the file it can read (TI.txt) and the one it cannot read (VMA.txt)

I am reading 20 values for testing (3 digits form 1 value )..
Is there any way I can modify this code to read values line by line..
I found a function called parseInt but I am not able to use it in my code..
Please help !!

v_m_a

As for buffering I tried to read values in double loop ..size of inner loop=63=assumed size of sector in SD CARD.. but inner loop reads only 1 value..I have attached the code alongside..

PaulS

Quote
My problem is values are stored n a text file line by line.. and I am not able to read them line by line..

So, you read one character at a time from the file. If that character is not a carriage return or line feed, store it in the next position in an array.

If it is a carriage return or line feed, process the data in the array, reset the array, and discard the character.
The art of getting good answers lies in asking good questions.

Go Up