Go Down

Topic: Storing large amount of integers? (Read 1 time) previous topic - next topic

bakenzilo

Hi,

I need to store for my program large amount of sensor readings ( 3 000 - 4 000) temporarily so I can do some analysis after that. I can't store them in RAM since it's only 1 KB big nor in EEPROM since it's only 512 KB big (I need 7-8 KB). So, flash memory would be appropriate but you can store there only static information (right?) so that doesn't help either. Any ideas? I'm really stuck here :((

Thanks

PaulS

Quote
but you can store there only static information (right?)

Flash memory is read only, and is populated by the loader when the sketch is compiled, linked, and uploaded. So, no that is not a viable option.

Quote
Any ideas?

Store the data on an SD card, or take fewer readings, process those, then take some more. Repeat as needed.

bakenzilo

Thanks, I suppose I'll have to go with fewer readings then, SD card is not an option

Cheers


robtillaart


What is the range of the values of the integers? does it fit in one byte  (0..255) ? would make 50% difference.

It is technically also possible to send the data over wifi / serial / I2C / Xbee etc to a storage server. is tha feasable?
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Mr_arduino


Thanks, I suppose I'll have to go with fewer readings then, SD card is not an option

Cheers


Why is an sd card not an option? Speed?
If so have you considered parallel flash memory?
Or maybe a cf card interface?
Are already using all your pins?
if you are already using the hardware spi then you can plug your device and the sd card and the device and use chip select to select between the device and the sd card.
if you are using all your pins and this is not because of already using the hardware spi have you conisdered an i/o expander?

noiz354

How do i read csv from sdcard ? as i know, it read certain pattern only

something look like this ,the modified code from sdfatlib :

Code: [Select]
sdin >> f1 >> c1

for 1,2,3,...,N

but it read only 1,2,3,... but in the last number it failed

do i have to read through byte , gain it to char[number of array] then convert from array of char to int

Graynomad

Just repeating what robtillaart asked, What is the range of the values of the integers?

______
Rob
Rob Gray aka the GRAYnomad www.robgray.com

noiz354

i'm want to use 16 bit int , number of integers : 256*256 , row*line

The csv is 2d array And want to read only 1 line for calculation , so only 256

Graynomad

So that's only 512 bytes then, stick 'em in RAM.

_____
Rob
Rob Gray aka the GRAYnomad www.robgray.com

bakenzilo



What is the range of the values of the integers? does it fit in one byte  (0..255) ? would make 50% difference.

It is technically also possible to send the data over wifi / serial / I2C / Xbee etc to a storage server. is tha feasable?


it seems we think alike :)

it doesn't fit but I made all values greater than 255 to be 255 so it does fit now, although it's not perfect solution but it helps :)

it's possible to send over Xbee but that's simply too slow for me, and I think SD card is also that's why I said it's not an option... I need to have a sensor reading at least (!) every 3-4 ms, is it possible to store data to SD card that fast?

but in the end I think I'll manage to store everything I need with byte array solution and less readings

robtillaart


Trunking values above 255 to 255 is a way to byte-ify the dataset. you keep the precission at cost of range

Scaling is another way, you keep the relative range at the cost of precission.

int maxValue = 768; // just an example
long x = readData(); // the long is to big to hold the int but

byte b = (x*255+127)/maxValue;

Please post your code, maybe there are more ideas...
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

CrossRoads

Or get a bigger uC:
'1284 has
4K Bytes EEPROM
16K Bytes Internal SRAM

make a big RAM buffer with that.
Designing & building electrical circuits for over 25 years.  Screw Shield for Mega/Due/Uno,  Bobuino with ATMega1284P, & other '328P & '1284P creations & offerings at  my website.

bakenzilo


Please post your code, maybe there are more ideas...


my code is still a bit messy, I'll do few changes in following days and inform you about progress and post code if needed

I like this scaling idea though, have to think about it, seems as better solution for me since I have to seperate high sensor readings from lower ones and it seems easier (more accurate) this way, hmmm

what I'm doing is sending bits (128bit key) over optical channel ("1" is light on and "0" is no light, actually better fits "less light" :D) and detecting them with arduino

p.s. appreciate all the help :)

bakenzilo


Or get a bigger uC:
'1284 has
4K Bytes EEPROM
16K Bytes Internal SRAM

make a big RAM buffer with that.


I'm doing this for my graduate thesis and already have 5 Arduino Unos that I need to use :D

AWOL

Quote
already have 5 Arduino Unos that I need to use

You could make a RAIA (Redundant Array of Inexpensive Arduinos)
"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

Go Up