Importing a large lookup table

In search for the way to import an array of numeric data from a separate file into my sketch, I found a post in a c++ forum that showcased this code:

static const char* data[] = {  // <--- new line 
#include "mydata.hh" 
};   // <--- don't forget semi-colon

I put in "mydata.hh" the following text:
"tizio", "caio", "sempronio"

and created the following sketch:

static const char* data[] = {  // <--- new line 
#include "E:\Downloads\Devices\Arduino\arduino-1.0.1\data\mydata.hh" 
};   // <--- don't forget semi-colon

void setup() {
  Serial.begin(9600);
}

void loop () {
  Serial.print (data[0]);
  Serial.println();
  
}

It compiles flawlessly but the serial monitor shows nothing (should show "tizio")!

Then I tried this:

static const int data[] = {  // <--- new line changed char* to int
#include "E:\Downloads\Devices\Arduino\arduino-1.0.1\data\numbers.hh" 
};   // <--- don't forget semi-colon

void setup() {
  Serial.begin(9600);
}

void loop () {
  Serial.print (data[0]);
  Serial.println();
  
}

with numbers.hh containing:
1,2,3,4

It also compiles flawlessly but the serial monitor shows nothing (in other attempt it showed numbers totally unrelated to the content of the file)!

I feel I am very close on learning how to import numeric data from an external file but I am unable to walk the final mile.

BTW, Arduino reference says that using pointers is very complicate and I probably don't need them anyways :frowning:

Any help is appreciated

TIA

How about posting the file? Or at least the first few lines of it.

It should work, are you sure the baud speed is the same in the serial monitor window?

How big is this file, anyway? How many numbers are in it?

It compiles flawlessly but the serial monitor shows nothing (should show "tizio")!

On what version of the IDE? Prior to 1.0.1, the failure of the compiler to find an include file was a fatal error. With 1.0.1, the failure to find an include file doesn't even generate a warning.

The failure to find the include file would explain the behavior you are seeing.

Eventually, my goal is to import a bidimensional array of numbers, but right now, for testing purposes I am experimenting with the following:
The entirety of "mydata.hh" contains the following text:

"tizio", "caio", "sempronio"

The entirety of "numbers.hh" contains:

1,2,3,4

"mydata.hh" on disk is 30 bytes long.
"numbers.hh"on disk is 9 bytes long.

guix:
It should work, are you sure the baud speed is the same in the serial monitor window?

Yes I made sure they are the same.

PaulS:
With 1.0.1, the failure to find an include file doesn't even generate a warning.
The failure to find the include file would explain the behavior you are seeing.

I am using 1.0.1
If I don't put the whole path, the compilers gives me an error: "No such file or directory".

Thanks for your help.

Ok, this is maddening!

This morning, after posting the reply above I re-run the sketches without changing anything and they work!

One thing that I did not mention is that yesterday, after changing something here and there and re-verifying I was getting a bunch of random errors that did not make much sense, such as that I already declared variables that were not even in the sketch anymore.....

Next I will actually try to import my large bidimensional array with the same method and I may reach for help again, but thank you for taking the time to answer.

One thing: does Arduino work better in daytime?
Any problem with full moon? :slight_smile:

I was getting a bunch of random errors

I've never heard of a compiler producing random errors.

I've never heard of a compiler producing random errors.

Incomprehensible, infuriating, annoying, yes. Random, no.

PaulS:

I've never heard of a compiler producing random errors.

Incomprehensible, infuriating, annoying, yes. Random, no.

Perhaps random was not the right term, incomprehensible is better.

I was getting something like (I don't remember it verbatim and the error did not show up again, this morning):
variable char[] already defined
when I never defined a variable called char[].
In a previous sketch I had defined
char* data[]

Maybe there is some reason for it {the pointer was still in the computer/arduino memory ??)
But if there is it is definitely not obvious :0

The entirety of "numbers.hh" contains:

That wouldn't be the right way to go.

You need two files:

numbers.h:

//header file for numbers

extern const unsigned char mydata[];

numbers.c/ino/pde

#include "numbers.h"

const unsigned char mydata[]={
  1, 2, 3, 4,
  ...
};

In your user application, you simply include numbers.h in order to reference mydata[].

Ok, I think I did it.
I found a way to keep a large array of data in a separate text file and importing it in my sketch as an #include.
The advantages of doing that are:

  • you can modify and update your data file without touching the sketch.
  • you don't have to put an unsightly long declaration in your code.
  • you don't even have to re-verify your code after modifying the data but you have to re-upload it

I thought this was a fairly established thing to do but I could find hardly any documentation.
I am posting my solution here hoping it may be helpful to some other newbie.

/* This examples illustrates how to import numeric data from a separate Text file

   By Luca Brigatti - 2012 
   witht he help of the folks in the Arduino programming forum.

   Note: Leaving the first dimension empty: data[][2]
   and using: sizeof(data) 
   to calculate the size of the array in bytes eliminates the need to know beforehand 
   how many values are there in the file.
   
   The operation: maxIndex = sizeof(data) / 4 - 1
   return the highest index number of the first dimension:  data[maxIndex][2]
   The denominator (4 in this case) is the product of the number of bytes per data type (2 bytes per integer) 
times the size of the second dimension ([2] in this case)
   
   The text file being read: "numbers2.hh"
   contains something like that: 
   ________________________
   // Example of bidimensional numeric data array
   // The actual numbers can be any int (if the data array is int)
  {1,1},
  {-2,4},
  {3,-9},
  {4,}, // Second element here will be 0, Like:  {4,0},
  {-5,-25},
......
_________________________________

Note: {,4},   i.e leaving the first place empty is NOT allowed.  
*/

// Main importing instructions. data[][2] can be a variable
static const int data[] [2] = {  
// Put the name of the file with the complete Path
#include "E:\Downloads\Devices\Arduino\arduino-1.0.1\data\numbers2.hh" 
};

int maxIndex ;
int arraySize ;

void setup() {
  
  Serial.begin(9600); // Initiate serial
  
  arraySize=sizeof(data);  // Gets the size of the array in bytes
  maxIndex = arraySize/4 - 1; // Size /2 (bytes in an int) /2 (for a bidimensional array) - 1 (as index starts at 0)   
  
  // Print dimensions of the array
  Serial.print("Array Size (bytes): ");
  Serial.println (arraySize);
  Serial.print ("# of rows: ");
  Serial.println(maxIndex+1);
  
  delay(2000);
}

void loop () {
  
  // Print all the data in the array, over and over again
  for (byte i=0; i<=maxIndex; i++) {
    Serial.print (data[i][0]);
    Serial.print (" , ");
    Serial.println (data[i][1]);
  }
  delay(500);
    Serial.println();
  
}
  • you don't even have to re-verify your code after modifying the data but you have to re-upload it

Yeah, right. Perhaps you want to enable verbose output when you compile, but not when you upload. Then, see what happens when you click Upload.

"I am posting my solution here hoping it may be helpful to some other newbie."

That approach works if the data is referenced from one source file.

If you want to reference the data from multiple files, you have to split it into .c and .h files.

If you want to reference the data from multiple files, you have to split it into .c and .h files.

Of course, it would be a lot easier if you used .cpp and .h files.

dhenry:
That approach works if the data is referenced from one source file.

Yes, that was exactly my goal:
ONE file, separate from the sketch itself, containing all my data.

Besides, nothing would prevent me to use another #include for a different set of data:

static const int data[] [2] = {  
#include "E:\DataPath\numbers2.hh" 
};

static const byte other_data[] [3] = {  
#include "E:\DataPath\other_numbers.hh" 
};

ONE file, separate from the sketch itself, containing all my data.

That's not the point.

Your approach wouldn't work if multiple files make wants to reference the data in "numbers.hh".

dhenry:

Your approach wouldn't work if multiple files make wants to reference the data in "numbers.hh".

Sorry, that I didn't understand.
Once I have my data in an array, I can reference that in the sketch, I won't need to reference "numbers.hh" anymore.

Importing a large lookup table

Thot:
"mydata.hh" on disk is 30 bytes long.
"numbers.hh"on disk is 9 bytes long.

Your idea of "large" and mine must differ.

It was probably a small test file just to demonstrate the problem... :slight_smile: