Newbie, store CSV (or its data) into Flash

Projects Programming
1

Newbie here, learning to program from zero.

I chose the Due for the 512 kB of Flash because I have a few hundred kB of LUTs. The LUTs is currently an excel file but I can store it as csv. When I search, I find countless examples of how to read and parse data from an SD card or hard code in the data into your sketch, like this:

const char image[504] = {
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe0, 0xf0,
  0xf0, 0xf8, 0xfc, 0xfc, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c,
  0x7c, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c, 0x7c,
  ...
  0x1e, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f,
  0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f,
  0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x1f, 0x0f,
  0x0f, 0x07, 0x01, 0x00
};

My Arduino don't have any SD card reader and I don't want to have it either. I rely on the ample Flash memory. It won't be user changeable anyway.

Two questions:

  1. How would I go about somehow appending or make the Arduino programming environment (IDE?) read a csv file which I have on my computer in the same folder as the sketch? Manually typing in the values is unthinkable. Parsing the data manually each time I need to tune something during development would be doable but I can't be the first which this issue. There has to be a simpler way.

  2. What would be a reasonable way to threat the data once in the Flash? PROGMEM? At runtime I only need to read my LUTs, never write or update.

2

Open the .csv file in an editor and use find/replace to get it into the format required to be put into an Arduino array then copy/paste it into the sketch

3

Thanks. I see how that could work, but is there really no way to just point to a CSV file?

A related next question is how to define that array since I have columns with Strings (name and description) and others with decimal numbers (actual data). The latter I can change to hex if that helps.

4

If you have mixed data types consider using an array of structs but I have no experience of using them in PROGMEM

Could you consider using a different board that would allow you to use SPIFFS ?

5

Consider instead of a Due, using an ESP32. 4Mb of FLASH, and you can use a FLASH filesystem to directly store the CSV files. ESP32 is a fantastic chip!

6

I'm not married to the Due in any way, but it could do hardware PWM fast enough, had enough pins and memory to store all my LUTs and was in stock.

7

Thanks. I will. How portable is the code between them?

8

Reasonably portable, but not 100%, depending on what you're doing. The ESP is MUCH faster, having two higher speed processors, and FAR more memory. Cost is about the same, and the ESP is smaller, has WiFi and bluetooth. It's primary shortcoming is more limited I/O. But it is my go-to chip for most projects. The Due is ok, but the available boards are spotty quality-wise. Some are absolute garbage. The ESP32 you can buy from the big electronics suppliers like Mouser and Digikey, about $20 each last I knew, and are actually built by Espressif, so quality is excellent.

9

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.