Defining array-length during setup

I have some data stored in a file. During setup the arduino should read this file, generating arrays from this data. Despite the fact that I haven't studied SD-Card-Usage I first would like to know how to handle the arrays.
Particulary the array-length is bothering me. How can I set it to the number of previously counted items?


Your description is a little vague, but I'd suggest you look at using malloc.

I have a file
1     0     0   0   255 #
2  255     0   0    0  #
3     0  255   0    0  #

and I need it to make some arrays:

#define rows 3
#define columns 4

int arr_ID[rows] = {1, 2, 3};
byte arr_col[columns][rows] = {{0, 255, 0}, {0, 0, 255}, {0, 0, 0}, {255, 0, 0}};

As the arrays do not change after setup I hoped to get around those dynamic arrays using malloc...

If you know the maximum size you can define a static array large enough (given the 2K SRAM an Arduino has)
Another option to consider is to write the data to EEPROM - - you can hold quite some values there, and if they don't change (too much) that might be an option too.


As the arrays do not change after setup I hoped to get around those dynamic arrays using malloc...

You can only statically define arrays at compile time. At run time, when setup() is reading the SD card, the array size is dynamic, so you must use malloc.

Or, allocate the array with static sizes large enough to hold all the data you will ever write to the file.

As the arrays are somehow pretty multidimensional they could soon get really big, that I already planned to store it into progmem.

This really got more of a problem than I tought. I really have to think about how to split from the table the needed array and just load this one into flash. Maybe I could optimize here and calculate a bit with spaces.

As I use the file to get some data from a computer to the arduino I might consider pre-compiling it into an array already. I think I read anywhere something about reading compiled code with arduino... But maybe this is too much...

thanks for the help so far.

Worst case, how many elements / dimensions are you talking about?

What does the data represent?
Are there symmetries in the dataset? (point/mirror/translation/etc)

There are techniques to compress arrays:
e.g. if you have a sin() lookup table, you only need to store from 0..90 degrees, because of point and mirror symmetries in the table.
And with that same table you can calculate the cos() , tan() and cotan() too. The price for the smaller table is in code, it must do the folding etc before lookup succeeds.

The main array is an array of a user-defined structure 'Program'

Each item of 'Program' contains an array with six columns. The first column contains a timestamp (long) while the others hold an int. So each row has 14bytes. The length of the column could vary between 0 to 1500 (but usually <100).

The 'main array' will hold 10-20 of those 'Programs'.

If I want to allow just one or two programs to reach the 1500 I have to lock a huge amount of space even for the other 10-20 programs.

My backup-solution would be to fill the array in the code, rewrite the code everytime my arrays change, recompile it and upload it on the arduino. But this would not be very nice...


You are aware of the <2 KB amount of total RAM in the Arduino, right? The only thing you're going to fit 1500 of at all is maybe bytes, if your code is small enough.

As I am using a AT2560 I should have 8kB right? Using the flash I get over 200kB for storage?

This should be enough for at least one 1500-liner. If I solve the problem to store the shorter arrays in a smaller spaces than the larger ones I should not get in trouble.
Besides 1500 is the absolute maximum. Most likely it is never needed, but if I have to define its maximum previously it should be 1500. I could set a limit to a other maximum, but it is hard for me to guess what should be enough. 300 could probably be enough. But if it someday shows that it is not, I have to reconstruct the whole thing.
So right now I am searching for the perfect solution. If there is no way to do it I can step back to either recompile the code everytime or shorten the arrays by force and hard limits.

Yes, you do have 4x more RAM than I do (I'm jealous :slight_smile: ) But my understanding is that FLASH can't be written to at runtime. Someone correct me if I'm mistaken...

As I am a really newbie myself I should not be the one spreading knowledge, but have a look here:

The first column contains a timestamp (long)

Depending on the max value of the timestamp you may need less than 4 bytes.
What is the difference between the minimum value and the maximum value?
--> you could encode the delta with previous timestamp..

1500 x 14 bytes = 21000 bytes ==> too much

So right now I am searching for the perfect solution

How do you expect the data to be processed? sequentially or random acces?

Sequentially: ==> Have you considered paging?

Just read the file partially in memory.
=> read N lines and if processed you read the next N lines (overwriting the buffer) and so on? N can be as small as 1 and as big as 100

Random Access : ==> If you don't know which part of the file you need, you could consider building a cache.

If line needed => search in cache
if avaialable in cache => use it
else fetch line from file
add to cache (optionally remove the least frequently used)
=> use it


Depending on the max value of the timestamp you may need less than 4 bytes.

right. I need 24h in seconds. So it is 86400 (2bytes).

I might optimize the 5 other columns as well. I need a value from 0-1000. That is 1000^5 = 1E+15 possibilities. A unsigned long can hold 2^16, so two of them could hold 1.8E+19 what should be enough.

But I think I really should shorten the maximum array. As it will work with most cases I should set the maximum to 300.
But still: 300 * 8 = 2400 bytes per program and I need at least 4 accessable and other 10-20 stored somewhere.

The arrays are too big for the RAM, but I could store them in the Flash, couldn't I?


but I could store them in the Flash, couldn't I?

Not unless you rewrite the bootloader.

right. I need 24h in seconds. So it is 86400 (2bytes).

2 bytes == 256*256=65536??

2 bytes == 256*256=65536??

right. this had not been the solution anyway.

my main problem still is that all arrays have to be of the maximum size. Most arrays just need <20 lines, just a few need more than 100 lines and maybe one above 1000.
If I don't want to save the 1000 lines for all arrays I need dynamic memory allocation as it seems. So I should read into this a bit...