Making an array of arrays

Hey! I'm trying to figure out how to make an array of arrays - not exactly a 2D array, but something more like this:

prog_uint8_t pattern1[] PROGMEM = {2, 14, 44, 95};
prog_uint8_t test[] PROGMEM = { &pattern1 };

Such that later on in my program, I can "randomly" access different arrays more easily. Any ideas?

prog_uint8_t multiarray[][] PROGMEM = {{1, 1, 1, 1}, {2, 2, 2, 2}};

This code works?

No, because the inner arrays will be of different sizes. The code I posted is just an example, my actual code will have maybe a dozen different 'inner' arrays with about 90 or so entries in each one.

h4t:
No, because the inner arrays will be of different sizes. The code I posted is just an example, my actual code will have maybe a dozen different 'inner' arrays with about 90 or so entries in each one.

Ok, you are in the right way with your sample.
You can store array pointers in array, but you will no be able to get the array size back.

This is ok, I will use sizeof or manually store another array of just the lengths to use :slight_smile:

But the code I posted does not work, I'm not sure what exactly to do. I admit I am from a web dev and Java background, so pointers and references are not my strong suit :smiley:

This will compile at least:

prog_uint8_t pattern1[] PROGMEM = {2, 14, 44, 95};
prog_uint8_t* test[] PROGMEM = { pattern1 };

Sizeof isn't going to do you much good though. Size of any of your array pointers will be 2. You'll have to store the sizes in an array somewhere as you suggested.

Neat, thank you!

I have a related question:
If I don't want to "ugly up" my main sketch file, can I store my super long arrays in another file, and include them into my main sketch? I have tried to make a new tab and just cut+paste the array definitions, but then when I try to reference them in my main sketch file, they are undefined / not in scope.

An illustration of the process (one way to do the deed):

// Demonstration of 2-D array in PROGMEM.
// It's not necessarily a "matrix" since
// each "row" can have a different number of
// elements ("colunmns").
//
// The principle is the same as would be used
// for dynamically-allocated 2-D arrays on
// a Princeton (Von Neumann) machine, but because
// of the Harvard architecure of the ATmega, we
// can't use the convenient 2-D notation that we
// all know and love so much.
//
// Oh, well...
// 
//
// davekw7x

#include <avr/pgmspace.h>

// A couple of typedefs for variables that
// will be in PROGMEM.
typedef float PROGMEM const prog_float;
typedef int   PROGMEM const prog_int;

// 2-D array is created by using an array of pointers.
// Each pointer is the address of a row of the 2-D array.
// I'll define a pointer to a float in PROGMEM.
// (Left as an exercise: Do it with a typedef instead of #define)
#define PFLT const prog_float *


// First define the "rows" of the 2-D array
PROGMEM prog_float row0[] = {
    1.23,
    4.56,
    PI
};

PROGMEM prog_float row1[] = {
    4.5,
    6.7
};


// Table of addresses of the "rows" of the 2-D array
PROGMEM prog_float * table[] = {
    row0, // Address of row0[0]
    row1  // Address of row1[0]
};
// For each row, these give the number of elements on each row
const int ROW0_SIZ = sizeof(row0)/sizeof(row0[0]);
const int ROW1_SIZ = sizeof(row1)/sizeof(row1[0]);


// This is a table of ints that give sizes of the individual "rows" of the 2-D array
PROGMEM prog_int rowsizes[] = {
    ROW0_SIZ,
    ROW1_SIZ
};

// This is the number of "rows"
const int TABLE_SIZ  = sizeof(table)/sizeof(table[0]);


void setup()
{
    Serial.begin(9600);    
}

void loop()
{
    // For each row
    for (unsigned i = 0; i < TABLE_SIZ; i++) {
        int tablei_siz = (int)pgm_read_word(rowsizes+i);
        for (unsigned j = 0; j < tablei_siz; j++) {
            PFLT pf = (PFLT)pgm_read_word(table+i);
            float x = pgm_read_float(pf+j);
            Serial.print("x[");
            Serial.print(i);
            Serial.print("][");
            Serial.print(j);
            Serial.print("] = ");
            Serial.println(x);
        }
        Serial.println();
    }
    Serial.println();
    delay(10000);
    Serial.println();
}

Output:


x[0][0] = 1.23
x[0][1] = 4.56
x[0][2] = 3.14

x[1][0] = 4.50
x[1][1] = 6.70

To get anything out of the table, you have to know where the table of row sizes is and how many rows there are. I mean, you could store the number of rows somewhere, but where will it end??? How much do you really need to know about the table?

Regards,

Dave

Thanks! I'm going to need to do a little more digestion, but I do have a couple questions so far:

  1. How does the following line give you the number of elements in a row? Can't you just use a single sizeof() command?
const int ROW0_SIZ = sizeof(row0)/sizeof(row0[0]);
  1. I didn't know you could just say "unsigned" without a datatype - does the compiler choose an appropriate datatype in this instance? ex:
for (unsigned i = 0; i < TABLE_SIZ; i++) {
  1. Since I'm not so experienced with memory manipulation, can I instead de-reference (with &) these pointers to the the value directly, instead of using pgm_read_word?

I will need to do some more brushing up on memory allocation and whatnot, but the code does compile just fine, so I just have to figure it out.

Also, do you have any tips on how to store long arrays in a different file, and bring it into Arduino for use? See my previous post in this thread for more details.

h4t:
...Can't you just use a single sizeof()

The sizeof operator always gives the number of bytes occupied by its operand. If the operand is the name of an array, the sizeof operator tells the number of bytes in the array. The number of elements in an array is equal to the number of bytes in the array divided by the number of bytes in an element of the array. (All of the elements of an array are of the same type and, therefore have the same size. I just use arrayname[0] to get the size of an element.)

h4t:
..."unsigned" without a datatype

In C and C++ declaring a variable to be unsigned always means that it is an unsigned int.

h4t:
can I instead de-reference (with &) these pointers to the the value directly, instead of using pgm_read_word?

No.

In C and C++ if x is a pointer data type and n is an integer data type the following two things mean the same to the compiler

x[i]

Exactly the same as

*(x+i)

Since the array is in PROGMEM, we have to use the one of the prog_read functions (or equivalent) to access an element of the array rather than just dereferencing the pointer. That is because program memory is in a totally different space, architecturally from "normal" data memory in this type of processor. That's why we go through all this agony. Sorry.

h4t:
I will need to do some more brushing ... just have to figure it out.

No one was born knowing this stuff, right? The Arduino originators and developers have labored mightily to make it possible for beginners (and a not necessarily insignificant number of not-so-beginners) to do certain things without going through all of the gruesome details that "real" C (and C++) programmers deal with.

Once you get into stuff like 2-D arrays in program memory of a Harvard architecture machine, well, the training wheels are off. You are also dealing with things above and beyond standard C and C++, since the C and C++ language definitions are done in such a way that they are deliberately ignorant of all hardware details like memory organization, files and file organization, uarts, terminals, etc., etc., etc. The language standards also don't specify any details of what commands are used (or how they are invoked) to create executable code from source. That means that even some experienced programmers have to learn new stuff when dealing with embedded systems and other applications not exactly like the workstations on which they learned programming.

h4t:
...how to store long arrays in a different file...more details.

With only a couple of (minor) changes to code that is already present (and working), here is a way to do what I think you are asking about. (See Footnote.)

1. Start a brand new sketch (keep the old one for reference in case you screw the pooch trying to reorganize the project).

2. Paste the following into the new main sketch window:

// Demonstration of 2-D array in PROGMEM.
// It's not necessarily a "matrix" since
// each "row" can have a different number of
// elements ("colunmns").
//
//
// Array definitions are in a separate C++ file that is linked
// in with this.
//
// This file needs access to the array of pointers to the "rows"
// of the 2-D arrayh and the array of ints that tells the
// sizes of the individual rows.  These are all given
// in the header file "array2D.h"
//
// davekw7x

#include <avr/pgmspace.h>
#include "array2D.h"

void setup()
{
    Serial.begin(9600);    
}

void loop()
{
    // For each row
    for (unsigned i = 0; i < TABLE_SIZ; i++) {
        int tablei_siz = (int)pgm_read_word(rowsizes+i);
        for (unsigned j = 0; j < tablei_siz; j++) {
            PFLT pf = (PFLT)pgm_read_word(table+i);
            float x = pgm_read_float(pf+j);
            Serial.print("x[");
            Serial.print(i);
            Serial.print("][");
            Serial.print(j);
            Serial.print("] = ");
            Serial.println(x);
        }
        Serial.println();
    }
    Serial.println();
    delay(10000);
    Serial.println();
}

3. Create new tab and name it "array2D.h"

4. Paste the following into the array2D.h tab:

// Header in tab named array2D.h
//
// davekw7x

#ifndef ARRAY2D_H__
#define ARRAY2D_H__
#include <WProgram.h>
#include <avr/pgmspace.h>

// A couple of typedefs for variables that
// will be in PROGMEM.
typedef float PROGMEM const prog_float;
typedef int   PROGMEM const prog_int;

// 2-D array is created by using an array of pointers.
// Each pointer is the address of a row of the 2-D array.
// I'll define a pointer to a float in PROGMEM.
// (Left as an exercise: Do it with a typedef instead of #define)
#define PFLT const prog_float *
extern PROGMEM prog_float * table[2];
extern PROGMEM prog_int rowsizes[2];
extern const int TABLE_SIZ;

#endif;

5. Create a new tab and name it array2D.cpp

6. Paste the following into the array2D.cpp tab:

// Code that defines and initializes 2-D array variables in
// a tab named array2D.cpp
//
// Demonstration of 2-D array in PROGMEM.
// It's not necessarily a "matrix" since
// each "row" can have a different number of
// elements ("colunmns").
//
// The principle is the same as would be used
// for dynamically-allocated 2-D arrays on
// a Princeton (Von Neumann) machine, but because
// of the Harvard architecure of the ATmega, we
// can't use the convenient 2-D notation that we
// all know and love so much.
//
// Oh, well...
// 
//
// davekw7x

#include "array2D.h"
// First define the "rows" of the 2-D array
PROGMEM prog_float row0[] = {
    1.23,
    4.56,
    PI
};

PROGMEM prog_float row1[] = {
    4.5,
    6.7
};


// Table of addresses of the "rows" of the 2-D array
PROGMEM prog_float * table[] = {
    row0, // Address of row0[0]
    row1  // Address of row1[0]
};
// For each row, these give the number of elements on each row
const int ROW0_SIZ = sizeof(row0)/sizeof(row0[0]);
const int ROW1_SIZ = sizeof(row1)/sizeof(row1[0]);


// This is a table of ints that give sizes of the individual "rows" of the 2-D array
PROGMEM prog_int rowsizes[2] = {
    ROW0_SIZ,
    ROW1_SIZ
};

// This is the number of "rows"
const int TABLE_SIZ  = sizeof(table)/sizeof(table[0]);

7. Compile and execute the sketch. You should get the same results as when everything was in the main sketch.

8. Ta-daa.

Regards,

Dave

Footnote:
Now, in the context of a simpler main sketch, I conclude that a better name for the number of rows in the 2-D array would probably be something like NUM_ROWS instead of the TABLE_SIZE nomenclature. I left it as it was so that very little editing of previous code would be required to illustrate how to break up the project into separate files. Note that some programmers use all upper-case identifiers for const variables as well as identifiers from #define directives. I have done that here, but that's not necessarily the "best" style. It depends on what the conventions (procedures and practices) are in your particular context. (Instructor or boss or whatever...)

Wow, thanks for the awesome response, lots of really useful info there!

Just out of curiosity, I changed all the variables from "PROGMEM" to "const" to get them to be stored in the main memory of the ATMega (just to see what that would do to the compiled program size). Just doing this actually reduced the file size by nearly 1kb! It seemed counter-intuitive at first, but it looks like the compiler is seeing that the avr/pgmspace.h file is not used anywhere, and therefore does not include it. Instant savings!

This means that I could avoid the pgm_read_word calls (I've only got a couple weeks to build my project, so the less I have to learn the better) and use dereferences. Now, with the variables all in the same memory space, can I do the following?

const byte pattern1[] = {2, 45, 22};
const byte pattern2[] = {4, 100, 59};
const byte* patterns[] = { pattern1, pattern2};

void loop() {
  const byte currentPattern = &patterns[0];
  Serial.println( currentPattern[2] );

  // Will this print out "22" as hoped?
}

const byte currentPattern = &patterns[0];
Serial.println( currentPattern[2] );

// Will this print out "22" as hoped?

No, it won't. currentPattern is not defined as an array or a pointer, so the [] operator can not be applied to it.

const byte *currentPattern = patterns[0];

will define a pointer to the start of the first array, to which the [] operator can be applied.

h4t:
... can I do the following?

Why not try it? In the time that it takes to post a request and wait for a (hopefully) helpful response, couldn't you just try it?

I respectfully suggest that, in addition to the learning experience of trying to debug buggy code, you might find it more time-effective than posting a request and waiting for a response (which may or may not be helpful).

I mean, it's always OK to ask (reallly---as far as I am concerned, it is always OK), and if you need help, then I think the following might be more time-effective:

  1. Post the code. The complete sketch that you tried, not just the part that you think is giving problems. If it's part of a big project, boil things down to a small (but complete) sketch that shows the problem behavior.

  2. If it gave a compiler error message that you didn't understand, then post the message. Paste the complete message into your post (don't paraphrase).

  3. If it compiled but didn't give the results you expect, the tell us exactly what you got and what you don't understand about the difference between what you got and what you expected.

Anyhow...

In your code, patterns is an array of pointers to byte. Each element of the code is the address of the first element in an array of bytes. That's OK. (See Footnote.)

const byte* patterns[] = { pattern1, pattern2};

patterns[0] is a pointer to byte. Its value is the address of the first element in the pattern1 array.
patterns[1] is a pointer to byte. Its value is the address of the first element in the pattern2 array.
So...

The following can't possibly work because it is trying to initialize a byte variable with the value of a pointer.

  const byte currentPattern = &patterns[0];

You can, however, define a pointer to byte and initialize its value to be equal to the value of patterns[0] by the following:

    const byte * currentPattern = patterns[0];

Now, currentPattern is pointing to the first element of the pattern1 array and currentPattern[2] will refer to element number 2 of that array, which is pattern1[2].

Finally, note that, if you give Serial.print a char or a byte to print,it will display the ASCII character corresponding to the value of its argument. I don't know (or care) what ASCII character corresponds to a decimal value of 22, but I am really, really sure that Serial.print(currentPattern[2] won't print 22

If you want to print the decimal value of currentPattern1[2], then you can tell Serial.print what you have in mind:

Serial.print(currentPattern[2], DEC);

Regards,

Dave

Footnote:
I know that some people think my posts are too wordy, especially when I repeat myself, but sometimes I just can't help it. Really. It's a curse.

So---here goes:

In C and C++ if x is a pointer data type and n is an integer data type, the following two notations are treated exactly the same by the compiler

p[i]

Treated exactly the same as

*(p + i)

Exactly the same. Not "pretty much the same." Not "essentially the same." Exactly. Note that I am not saying that pointers are the same as arrays. Pointer variables positively, absolutely, unequivocally are not the same as arrays. That statement is incontrovertible. (And I am unanimous in that.) It's just notation.

Finally, it is important to remember that the name of an array is a const pointer to the data type of the declared array. The value of the pointer is equal to the address of the first element of the array.

Therefore, as far as the compiler is concerned, the following two statements are identical in the eyes of the compiler

const byte* patterns[] = {pattern1, pattern2};

Same as

const byte * patterns[] = {&pattern1[0], &pattern2[0]};

Many C and C++ programmers of my acquaintance seem to prefer the first form (less typing and less visual clutter, don'tcha know), but they mean exactly the same thing. Use whatever form seems appropriate to your context.

Found this threat useful/interesting

EVP:
Found this threat useful/interesting

Most people find threats intimidating rather than useful. Threads, on the other hand...

haha aarr.. yes the spell checker is usually a good thing. The odd one slip's by though