Importing a large lookup table

"I am posting my solution here hoping it may be helpful to some other newbie."

That approach works if the data is referenced from one source file.

If you want to reference the data from multiple files, you have to split it into .c and .h files.

If you want to reference the data from multiple files, you have to split it into .c and .h files.

Of course, it would be a lot easier if you used .cpp and .h files.

dhenry:
That approach works if the data is referenced from one source file.

Yes, that was exactly my goal:
ONE file, separate from the sketch itself, containing all my data.

Besides, nothing would prevent me to use another #include for a different set of data:

static const int data[] [2] = {  
#include "E:\DataPath\numbers2.hh" 
};

static const byte other_data[] [3] = {  
#include "E:\DataPath\other_numbers.hh" 
};

ONE file, separate from the sketch itself, containing all my data.

That's not the point.

Your approach wouldn't work if multiple files make wants to reference the data in "numbers.hh".

dhenry:

Your approach wouldn't work if multiple files make wants to reference the data in "numbers.hh".

Sorry, that I didn't understand.
Once I have my data in an array, I can reference that in the sketch, I won't need to reference "numbers.hh" anymore.

Importing a large lookup table

Thot:
"mydata.hh" on disk is 30 bytes long.
"numbers.hh"on disk is 9 bytes long.

Your idea of "large" and mine must differ.

It was probably a small test file just to demonstrate the problem... :slight_smile:

guix:
It was probably a small test file just to demonstrate the problem... :slight_smile:

Indeed :slight_smile:
I made up a couple of quick text files, one with strings, the other with integers, to see if I could make the darn thing work!

And when I finally made it to work (again, I did not do anything different from last night, today it just worked) it also works with the "real" thing.

It turns out that I was able to reduce the actual table, by organizing the data in a different way to a mere 400 bytes or so, well below the RAM restrictions.

Small size still, probably by your (and my) standards. :stuck_out_tongue:

You might be able to put the table into PROGMEM to minimize your RAM usage.

Also, the compiler tries to not compile things it doesn't think have changed. I'm not sure, but it's possible that if you change the .h file it won't realize to recompile the .cpp file. In other words, if you change the numbers but not the sketch.

I don't like #includes in the middle of a declaration, personally. Why not just put the whole array into a separate file (including the declaration, and the trailing brace)?

I thought of doing something like that but I don't know how to do it :cold_sweat:
Even thought of creating a "Library" but when I consulted the reference on how to do that I thought it was a bit of an overkill for my purpose.

Also, my way (well, the one I learned form a c++ forum) does not require two files (.h and .cpp) but just one: .hh (and by the way, I have no clue of the meaning of the .hh extension ....) which would seem to me more practical.

But again, I don't have much experience and I would not be surprised if there were a better way. :slight_smile:

Come to think of it, is the .h and .cpp business the same as creating a library?

Just put all the stuff in a separate .h file, eg. from my bootloader project:

byte PROGMEM atmega328_optiboot [] = {

0x11, 0x24, 0x84, 0xB7, 0x14, 0xBE, 0x81, 0xFF, 0xF0, 0xD0, 0x85, 0xE0, 0x80, 0x93, 0x81, 0x00, 

... blah blah ...

0xFF, 0x27, 0x09, 0x94, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x04, 0x04, 

};  // end of atmega328_optiboot

I don't know how to do it

I posted an example for you earlier.

You should never declare a variable in a .h file - that's so un-c that it will get you fired in most places. Declaring a variable in .h files creates localized global variables at each inclusion of those .h files.

Instead, declare the variable in the .c file and if such variables are to be referenced somewhere else, "extern" such variables.

@Nick

So, if I understand correctly:

In a separate file called mydata.h I put:

const int data [] [2] = {
     {1,2},
     {3,4},
     {5,6},
// and so on...

} ;

(assuming it's small enough to fit in RAM)

And then in the main sketch I put:

#include "mydata.h" ;

Is it that simple?
That would have the additional advantage that I can put several arrays in the same file (if I need them) and just use an #include in the main sketch.

What about the mentioned extra .cpp file?

dhenry:
I posted an example for you earlier.

You should never declare a variable in a .h file - that's so un-c that it will get you fired in most places. Declaring a variable in .h files creates localized global variables at each inclusion of those .h files.

Instead, declare the variable in the .c file and if such variables are to be referenced somewhere else, "extern" such variables.

Yes, thank you. I saw your earlier post but I did not fully understand that, sorry :slight_smile:

So, the example I just asked Nick about is not correct because I AM declaring the variable in an .h file?

Thanks

P.s. I'm glad I am not a programmer, finding a job these days is a pain ....

dhenry has got a point, however his suggested change has its own issues.

Let's make an example.

Sketch:

void setup ()
  {
  Serial.begin (115200);
  Serial.println (data [0] [0]);  
  }  // end of setup
  
void loop ()  {  }

foo.cpp:

const int data [] [2] = {
     {1,2},
     {3,4},
     {5,6},
// and so on...
} ;

Error:

sketch_nov22a.cpp: In function 'void setup()':
sketch_nov22a:5: error: 'data' was not declared in this scope

So the variable "data" exists, the sketch just doesn't know about it.

So we add an "extern" to the main sketch:

extern const int data [] [2];

void setup ()
  {
  Serial.begin (115200);
  Serial.println (data [0] [0]);  
  }  // end of setup
  
void loop ()
  {
    
  }  // end of loop

Still an error:

sketch_nov22a.cpp.o: In function `setup':
sketch_nov22a.cpp:10: undefined reference to `data'
sketch_nov22a.cpp:10: undefined reference to `data'

Different error (a linker error, not a compiler error).

The problem is that "const" data is not exported. So remove the word "const" from both files, and it compiles.

But now we ask ourselves, "how big is data"? Well if it was declared in the same file, we can find that out with sizeof. A helpful macro does that:

extern int data [] [2];

// number of items in an array
#define NUMITEMS(arg) ((unsigned int) (sizeof (arg) / sizeof (arg [0])))

void setup ()
  {
  Serial.begin (115200);
  
  for (int i = 0; i < NUMITEMS (data); i++)
    Serial.println (data [i] [0]);  
  }  // end of setup
  
void loop () { }

Unfortunately the compiler doesn't know how big data is, because it is in a different file, hence:

sketch_nov22a.cpp: In function 'void setup()':
sketch_nov22a:10: error: invalid application of 'sizeof' to incomplete type 'int [][2]'

If we make the extern an explicit size it will compile:

extern int data [3] [2];

// number of items in an array
#define NUMITEMS(arg) ((unsigned int) (sizeof (arg) / sizeof (arg [0])))

void setup ()
  {
  Serial.begin (115200);
  
  for (int i = 0; i < NUMITEMS (data); i++)
    Serial.println (data [i] [0]);  
  }  // end of setup
  
void loop () { }

But we have to keep the "extern" in step with the actual variable size in the other (.cpp) file.

@dhenry: In my particular example, I think the use of the .h file is OK. It works around the issue I described (not knowing the variable size) and in this case the include files were only included once. Plus, I won't be firing myself. :wink:

So here is how it works in your case:

numbers2.h (you can name it whatever you want, as long as it is consistent)

//hearder file for data[]
//to be included in user code

extern const int data[] [2];

numbers2.c

#include "numbers2.h"

const int data[] [2]={
    {1,1},
  {-2,4},
  {3,-9},
  {4,5}, // Second element here will be 0, Like:  {4,0},
  {-5,-25}
};

Your user code: a simple reference to data[]

#include "numbers2.h"

void setup(void) {
}

void loop(void) {
  unsigned char tmp;
  
  //an example of invoking data[];
  tmp+=data[0][(tmp & 0x01)?1:0];
}

numbers2.h can be referenced as many times as you want.

This is fairly standard C.

You haven't addressed the question: how do you know how big the array is?

Thanks guys,
I read your posts several times and I think I understand your solutions.

While I am not arguing that "my" solution is either more elegant or more proper, I am arguing it is more practical because:

  1. It involves one data file (.hh) instead of two (.h and .c)
  2. can be used with variables and constants
  3. allows the first dimension to be [] and therefore the sketch can calculate the size at execution time without knowing it at coding time.
  4. No need to keep the three files synchronized (that's really a corollary of #3).

Please correct me if I am wrong.

Thanks again, I'm learning a lot.