Pages: [1] 2   Go Down
Author Topic: Storing large amount of integers?  (Read 987 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 26
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi,

I need to store for my program large amount of sensor readings ( 3 000 - 4 000) temporarily so I can do some analysis after that. I can't store them in RAM since it's only 1 KB big nor in EEPROM since it's only 512 KB big (I need 7-8 KB). So, flash memory would be appropriate but you can store there only static information (right?) so that doesn't help either. Any ideas? I'm really stuck here smiley-sad(

Thanks
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 553
Posts: 46291
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
but you can store there only static information (right?)
Flash memory is read only, and is populated by the loader when the sketch is compiled, linked, and uploaded. So, no that is not a viable option.

Quote
Any ideas?
Store the data on an SD card, or take fewer readings, process those, then take some more. Repeat as needed.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 26
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks, I suppose I'll have to go with fewer readings then, SD card is not an option

Cheers

Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 170
Posts: 12487
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


What is the range of the values of the integers? does it fit in one byte  (0..255) ? would make 50% difference.

It is technically also possible to send the data over wifi / serial / I2C / Xbee etc to a storage server. is tha feasable?
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Offline Offline
Full Member
***
Karma: 6
Posts: 225
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks, I suppose I'll have to go with fewer readings then, SD card is not an option

Cheers


Why is an sd card not an option? Speed?
If so have you considered parallel flash memory?
Or maybe a cf card interface?
Are already using all your pins?
if you are already using the hardware spi then you can plug your device and the sd card and the device and use chip select to select between the device and the sd card.
if you are using all your pins and this is not because of already using the hardware spi have you conisdered an i/o expander?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 4
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

How do i read csv from sdcard ? as i know, it read certain pattern only

something look like this ,the modified code from sdfatlib :

Code:
sdin >> f1 >> c1

for 1,2,3,...,N

but it read only 1,2,3,... but in the last number it failed

do i have to read through byte , gain it to char[number of array] then convert from array of char to int
Logged

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8460
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Just repeating what robtillaart asked, What is the range of the values of the integers?

______
Rob
Logged

Rob Gray aka the GRAYnomad www.robgray.com

Offline Offline
Newbie
*
Karma: 0
Posts: 4
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

i'm want to use 16 bit int , number of integers : 256*256 , row*line

The csv is 2d array And want to read only 1 line for calculation , so only 256
Logged

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8460
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

So that's only 512 bytes then, stick 'em in RAM.

_____
Rob
Logged

Rob Gray aka the GRAYnomad www.robgray.com

Offline Offline
Newbie
*
Karma: 0
Posts: 26
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


What is the range of the values of the integers? does it fit in one byte  (0..255) ? would make 50% difference.

It is technically also possible to send the data over wifi / serial / I2C / Xbee etc to a storage server. is tha feasable?

it seems we think alike smiley

it doesn't fit but I made all values greater than 255 to be 255 so it does fit now, although it's not perfect solution but it helps smiley

it's possible to send over Xbee but that's simply too slow for me, and I think SD card is also that's why I said it's not an option... I need to have a sensor reading at least (!) every 3-4 ms, is it possible to store data to SD card that fast?

but in the end I think I'll manage to store everything I need with byte array solution and less readings
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 170
Posts: 12487
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Trunking values above 255 to 255 is a way to byte-ify the dataset. you keep the precission at cost of range

Scaling is another way, you keep the relative range at the cost of precission.

int maxValue = 768; // just an example
long x = readData(); // the long is to big to hold the int but

byte b = (x*255+127)/maxValue;

Please post your code, maybe there are more ideas...
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Global Moderator
Boston area, metrowest
Offline Offline
Brattain Member
*****
Karma: 444
Posts: 23861
Author of "Arduino for Teens". Available for Design & Build services. Now with Unlimited Eagle board sizes!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Or get a bigger uC:
'1284 has
4K Bytes EEPROM
16K Bytes Internal SRAM

make a big RAM buffer with that.
Logged

Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

Offline Offline
Newbie
*
Karma: 0
Posts: 26
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Please post your code, maybe there are more ideas...

my code is still a bit messy, I'll do few changes in following days and inform you about progress and post code if needed

I like this scaling idea though, have to think about it, seems as better solution for me since I have to seperate high sensor readings from lower ones and it seems easier (more accurate) this way, hmmm

what I'm doing is sending bits (128bit key) over optical channel ("1" is light on and "0" is no light, actually better fits "less light" smiley-grin) and detecting them with arduino

p.s. appreciate all the help smiley
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 26
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Or get a bigger uC:
'1284 has
4K Bytes EEPROM
16K Bytes Internal SRAM

make a big RAM buffer with that.

I'm doing this for my graduate thesis and already have 5 Arduino Unos that I need to use smiley-grin
Logged

Global Moderator
UK
Offline Offline
Brattain Member
*****
Karma: 241
Posts: 24480
I don't think you connected the grounds, Dave.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
already have 5 Arduino Unos that I need to use
You could make a RAIA (Redundant Array of Inexpensive Arduinos)
Logged

"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

Pages: [1] 2   Go Up
Jump to: