Importing a large lookup table

Hello,
I need to setup a large (100 or so pairs of ints) lookup table to connect two set of numbers that are not mathematically related, i.e. One cannot be calculated from the other.
I am plannig to set up an array like this:

int table [100] [1] ;

And then fill it like this:

table [0] [0] = x0 ;
table [0] [1] = y0 ;
table [1] [0] = x1;
table [1] [1] = y1;
........................

I could easilly do it within the sketch but that would make the sketch heavy to read.
Is there a way to set up the table in a separate text file:

xo,yo
x1,y1
x2,y2
........

and then read or import it into the array?

That would have the additional advantage to be able to tweak the table without modifying verifying and re-uploading the sketch

TIA

Put it in Flash memory of controller

If you have the hardware needed to read from an SD card, it would be relatively easy to read your data from a text file at run time. Alternatively you could store it in EEPROM, but you'd have to write a program on the PC to send the data to put there, both for the initial load and when you wanted to change it. Or you could put the initialization code in another tab in the IDE to keep it segregated from your main sketch. With this last method, you would still have to reload the sketch every time you made a change. Another way would be to use the Arduino as a web client and go get the data from a web server at startup.

Put the array itself in a .c file and declare it as extern in a .h file.

In your code, include the .h file and then include the .c file in the project.

That would have the additional advantage to be able to tweak the table without modifying verifying and re-uploading the sketch

EEPROM Or SD card best options.

int table [100] [1] ;``table [0] [1] = y0 ;
Sorry, no.

int table [100] [2] ;

(100 or so pairs of ints) lookup table

questions pop up.
What do these values represent?
what is the range of values?
Can you post a part of the table?
Would it fit in a byte ?
Are there repeating patterns in the table?
Are they interpolatable?

Are you using the tabs feature of the IDE?
Put the array and your other pin mappings and global variations in one tab, called a_presetup. Be sure to use progmem for the array unless the sketch needs to change the values.
put void setup in a 2nd called b_setup
put void look in a 3rd called c_loop
seperates things nicely.
When you save, the first tab will be your sketch file name, I use that for any notes I want to keep.

Thanks for all the answers !
I was (vaguely) thinking along the lines of dhenry and crossroads' solutions. I am going to research those
I was thinking maybe I can just #include it or something.
Another thing i thought would be to create a library that does the data loading but that seems to me more trouble that it's worth, to keep the long list away from the program itself.

I was thinking maybe I can just #include it or something.

You can. If you do that, you can only include it once, or you create local copies of global variables.

int table [100] [1] ;

As AWOL was suggesting, a table dimension of 1 isn't much use.

table [0] [0] = x0 ;
table [0] [1] = y0 ;
table [1] [0] = x1;
table [1] [1] = y1;

No. Assign the values like this:

int table [100] [2] = {
  { 1, 2 },
  { 3, 4 },

// and so on

  };

And then keep it in flash memory with the PROGMEM directive so it doesn't use any RAM.

eg.

int PROGMEM table [100] [2] = {
  { 1, 2 },
  { 3, 4 },
  { 5, 6 },

// and so on

  };

Thanks AWOL and Nick,
I am still baffled by the fact that in the declaration "1" means 1 total and in the assignment "1" means the second item: 0 - 1.
I keep on making this mistake and the IDE keeps on reminding me about that (I'm glad the IDE didn't get tired of that yet).

Thanks also for the PROGMEM tip and for the link! I can do it so long as the program does not need to modify it, correct?
Though it may not be necessary as an int array of 100 x 2 entries should occupy 2 x 100 x 2 = 400 bytes).
As the second numbers are not bigger than a byte I could create two monodimensional arrays, one int and one byte for a total of 100 * (2 + 1) = 300 bytes (plus some overhead, I assume).

Finally, can I put the code in a function at the end of the sketch and call it from setup(), maybe by stating somehow it's a global array or, being it a variable assignment must it go before setup() (that would make the sketch neater) ?

Thanks

Thot:
I am still baffled by the fact that in the declaration "1" means 1 total and in the assignment "1" means the second item: 0 - 1.

You declare the number you want. eg.

int foo [2];

You get an array of two. However they are zero-relative. So you index into 0, 1.

Thanks also for the PROGMEM tip and for the link! I can do it so long as the program does not need to modify it, correct?

Yes.

Finally, can I put the code in a function at the end of the sketch and call it from setup(), maybe by stating somehow it's a global array or, being it a variable assignment must it go before setup() (that would make the sketch neater) ?

The assignments would take instructions. It seems much simpler to pre-assign the array (ie. declare the contents, don't assign them). Then it would need to be globally defined (or at least, static inside a function).

As the second numbers are not bigger than a byte I could create two monodimensional arrays, one int and one byte for a total of 100 * (2 + 1) = 300 bytes (plus some overhead, I assume).

You could do that, but if you keep them in program memory you have 32 Kb to play with, so you aren't running out yet.

In search for the way to import an array of numeric data from a separate file into my sketch, I found a post in a c++ forum that showcased this code:

static const char* data[] = {  // <--- new line 
#include "mydata.hh" 
};   // <--- don't forget semi-colon

I put in "mydata.hh" the following text:
"tizio", "caio", "sempronio"

and created the following sketch:

static const char* data[] = {  // <--- new line 
#include "E:\Downloads\Devices\Arduino\arduino-1.0.1\data\mydata.hh" 
};   // <--- don't forget semi-colon

void setup() {
  Serial.begin(9600);
}

void loop () {
  Serial.print (data[0]);
  Serial.println();
  
}

It compiles flawlessly but the serial monitor shows nothing (should show "tizio")!

Then I tried this:

static const int data[] = {  // <--- new line changed char* to int
#include "E:\Downloads\Devices\Arduino\arduino-1.0.1\data\numbers.hh" 
};   // <--- don't forget semi-colon

void setup() {
  Serial.begin(9600);
}

void loop () {
  Serial.print (data[0]);
  Serial.println();
  
}

with numbers.hh containing:
1,2,3,4

It also compiles flawlessly but the serial monitor shows nothing (in other attempt it showed numbers totally unrelated to the content of the file)!

I feel I am very close on learning how to import numeric data from an external file but I am unable to walk the final mile.

BTW, Arduino reference says that using pointers is very complicate and I probably don't need them anyways :frowning:

Any help is appreciated

TIA

How about posting the file? Or at least the first few lines of it.

It should work, are you sure the baud speed is the same in the serial monitor window?

How big is this file, anyway? How many numbers are in it?

It compiles flawlessly but the serial monitor shows nothing (should show "tizio")!

On what version of the IDE? Prior to 1.0.1, the failure of the compiler to find an include file was a fatal error. With 1.0.1, the failure to find an include file doesn't even generate a warning.

The failure to find the include file would explain the behavior you are seeing.

Eventually, my goal is to import a bidimensional array of numbers, but right now, for testing purposes I am experimenting with the following:
The entirety of "mydata.hh" contains the following text:

"tizio", "caio", "sempronio"

The entirety of "numbers.hh" contains:

1,2,3,4

"mydata.hh" on disk is 30 bytes long.
"numbers.hh"on disk is 9 bytes long.

guix:
It should work, are you sure the baud speed is the same in the serial monitor window?

Yes I made sure they are the same.

PaulS:
With 1.0.1, the failure to find an include file doesn't even generate a warning.
The failure to find the include file would explain the behavior you are seeing.

I am using 1.0.1
If I don't put the whole path, the compilers gives me an error: "No such file or directory".

Thanks for your help.