Using an Uno to play back somewhat large table of analog inputs, best way?

Using Arduino Project Guidance
1

Hello!

I'm just starting to sink my teeth into Arduino for the first time so if there are noob questions here I apologize.

My project is to play back the sensor outputs from an engine over a certain dyno cycle. I'm using a cool component called a CANalyzer to output CAN messages for engine speed, etc, but that is completely separate from the arduino project (I'm just planning to press start on the CANalyzer and on the Arduino at the same time). On arduino I'm simply playing back 4 analog 0-5V signals from a MCP4728 (4 channel DAC) using I2C from an Arduino Uno. I found some libraries from the internet for it, and figured out how to update all the libraries I found for 1.05 IDE which is what I have. I've got that part solved and have the code to the point where I can output whatever voltage I type in, but that's just a single constant value.

I know the memory is very limited on the Arduino so I'm trying to figure out the best way to do the actual project, which is to play back the 4 signals over 1200 seconds, one point per signal each second.

So I guess I could make that a 1200x4 array and assume the index is the timestamp in seconds so I don't need a timestamp column.

So the first most basic question is how/where should that 1200x4 relay be stored, if it's even possible to do so? It will be composed of ints. Can I put a 1200x4 array of ints on there as a variable without killing the memory?

Failing that, another option would be to put 1200 lines of code, each of which manually changes the register, so the values are not stored as a variable but stored as code. Can the arduino handle 1,200 lines of code? This seems like a horrible way to do this but oh well.

Alternately if neither of these approaches work, I could either have the computer connected and send the data over serial somehow so it's not stored on the arduino, via Processing or otherwise (not ideal for my computer to be part of the system but if it's required it's fine), or include an SD card shield and have it read the CSV off the SD card (adds more complication and have to wait to order the shield).

Thanks for any input, we'll go from there. :slight_smile:

2

1200 x 4 is 4800; ints take two bytes, for a total memory requirement of 9600 bytes. You can't store that in RAM on an Uno. You can store it in program memory, if you have that much to spare. Here's a description of how to store and fetch data in flash: PROGMEM - Arduino Reference

3

'1284P has 16K SRAM, could store it there if the data needs to be varied by the code - or captured & played back.
This duemilanove style 1284P board I offer can do that (duemilanove style because it uses FTDI chip vs 8U2/16U2 chip for USB/Serial). The USB/Serial can be on-board as pictured here, or an off-board FTDI Basic(or similar) can be connected to a header that would be installed in place of the module.
I recently used it in a project with a 14500 byte array to do similar, I blasted the data out at 8 MHz rate into a large bank of shift registers.
http://www.crossroadsfencing.com/BobuinoRev17/

1094×458 70.8 KB

4

Thanks for the input so far. Thanks tmd3 for showing me exactly how to figure out how much space I'd be taking up.

Now I just need to figure out how to convert a CSV into an array and store it in program memory...

5

I know the memory is very limited on the Arduino so I'm trying to figure out the best way to do the actual project, which is to play back the 4 signals over 1200 seconds, one point per signal each second.

Personally, I would just send the data off the Arduino (use a Leonardo 32U4) too a PC over the USB. Collect the data in Excel.

Excel can then parse, analyze, graph, ...

Here is how to parse using formulas... No VBA required!
http://www.hackster.io/rayburne/arduino-to-excel-using-v-usb

Ray

6

jcaserta:
Now I just need to figure out how to convert a CSV into an array and store it in program memory...

To initialize an array, either in RAM or with PROGMEM, you put the values inside curly brackets, and separate them by commas. CSV stands for, "comma separated values." Does that suggest anything to you?

7

Where is the data ultimately coming from? If it's being acquired at the PC then I would be inclined to have the playback controlled there as well, and write a small PC app that just pulled values from the data file in whatever format you chose to store them, and wrote them to the Arduino serial port; the Arduino sketch would just take data sets from the serial port as they arrive and write them to the DACs.

CSV would be a reasonable choice for the file format and also for the Arduino serial comms, unless the data is already encoded in some other format.

8

tmd3:

jcaserta:
Now I just need to figure out how to convert a CSV into an array and store it in program memory...

To initialize an array, either in RAM or with PROGMEM, you put the values inside curly brackets, and separate them by commas. CSV stands for, "comma separated values." Does that suggest anything to you?

I am near certain the Op knows what a CSV is! My impression is s/he was just talking out loud. Your statement that I bolded above is not helpful., IMO.

Ray

9

mrburnette:

tmd3:

jcaserta:
Now I just need to figure out how to convert a CSV into an array and store it in program memory...

To initialize an array, either in RAM or with PROGMEM, you put the values inside curly brackets, and separate them by commas. CSV stands for, "comma separated values." Does that suggest anything to you?

I am near certain the Op knows what a CSV is! My impression is s/he was just talking out loud. Your statement that I bolded above is not helpful., IMO.

Yes, you are correct that I am very familiar with CSVs, thank you. It's arduino that's new to me. But if I'm making something way more complicated than it needs to be I'd love it if you told me that but again it's a 1200x4 array. So the CSV looks like:
1,100,110,100,2000
2,120,115,110,2050
3,125,113,112,2100
etc.

In order to put it into the code don't I need to add curly brackets into there and extra commas and stuff? So:
int myWayTooLongArray[1200][4]={
{1,100,110,2000},
{2,120,115,110,2050},
{3,125,113,112,2100},
};

Except that it's 1200 rows long so there's no way I'm converting from CSV to the second format manually like you're suggesting. Of course I could read it in as text programmatically in another language and have it output text with the curly brackets and stuff and that may be what I do but I figure there's probably an easier way. That's not something I can readily do - I'm very good with VBA and Excel but that's more of a normal text reading type function to reformat the text like that.

10

PeterH:
Where is the data ultimately coming from? If it's being acquired at the PC then I would be inclined to have the playback controlled there as well, and write a small PC app that just pulled values from the data file in whatever format you chose to store them, and wrote them to the Arduino serial port; the Arduino sketch would just take data sets from the serial port as they arrive and write them to the DACs.

CSV would be a reasonable choice for the file format and also for the Arduino serial comms, unless the data is already encoded in some other format.

Thanks for the input. I have a CSV of the data that I want to put into this thing, on my PC, that's where the data is from. I only have to put this one data set in, there's not currently a need to pull in other data sets on the fly. Originally it was from an engine but that part's already done, it's a nice 1200x4 CSV of data at one second intervals.

What you're suggesting was one of the options I originally mentioned, and it's still on the table. The reason I'm hoping to do it on the arduino standalone is because this thing is going to be sent out and used elsewhere and I'd prefer not to have to install arduino software on whatever computer it's being used with. It seems like doing that will be feasible with progmem, just have to figure out the best way to get the data into the arduino.

11

mrburnette:

I know the memory is very limited on the Arduino so I'm trying to figure out the best way to do the actual project, which is to play back the 4 signals over 1200 seconds, one point per signal each second.

Personally, I would just send the data off the Arduino (use a Leonardo 32U4) too a PC over the USB. Collect the data in Excel.

Excel can then parse, analyze, graph, ...

Here is how to parse using formulas... No VBA required!
http://www.hackster.io/rayburne/arduino-to-excel-using-v-usb

Ray

I'm not sure you understand the project. Firstly I already have an Uno so would prefer to use that unless it absolutely won't work. Secondly I'm trying to get the data from Excel onto the Arduino, not the other way around. Sorry I wasn't clear.

12

econdly I'm trying to get the data from Excel onto the Arduino, not the other way around.

Check into Serial.parseInt() and Serial.parseFloat() ... These can be used in a loop() to load an array. This array can be SRAM or in EEPROM.
Nice part about the Serial.parse... Functions is it "eats" the comma separator... No need to manage it in your code.

Ex from my calculator program: http://www.hackster.io/rayburne/scientific-calculator

 // ADD
          a = Serial.parseFloat();
          b = Serial.parseFloat();
          LastX = a + b;
          Serial << _FLOAT(LastX, decimals);
          break;
13

mrburnette:

econdly I'm trying to get the data from Excel onto the Arduino, not the other way around.

Check into Serial.parseInt() and Serial.parseFloat() ... These can be used in a loop() to load an array. This array can be SRAM or in EEPROM.
Nice part about the Serial.parse... Functions is it "eats" the comma separator... No need to manage it in your code.

Ex from my calculator program: http://www.hackster.io/rayburne/scientific-calculator

 // ADD

a = Serial.parseFloat();
          b = Serial.parseFloat();
          LastX = a + b;
          Serial << _FLOAT(LastX, decimals);
          break;

Will definitely look into that! Thanks

14

You only have 1K of EEPROM on an arduino. To make a stand alone system you need an external EEPROM chip. You then have two programs one to read data from the serial port from processing and write to the EEPROM and then you program the arduno with the other to do the replaying bit.
You can get I2C EEPROMs so they will sit nicely on the same bus as your D/A.

15

mrburnette:
Your statement that I bolded above is not helpful., IMO.

Apparently not. I thought it hinted at a solution to the OP's stated problem. Since it didn't seem to work, I'll be explicit this time.

Addressing the OP's directly stated concerns,

jcaserta:
Now I just need to figure out how to convert a CSV into an array and store it in program memory...

jcaserta:
... it's 1200 rows long so there's no way I'm converting from CSV to the second format manually ...

In view of the resources that he seems to have in hand,

jcaserta:
... I'm trying to get the data from Excel onto the Arduino ...

Here's how I'd put that data into a PROGMEM array:

  • Open the CSV file in a text editor, and copy it to the clipboard.
  • Open a spreadsheet, and paste. The reason I wouldn't open the CSV in a spreadsheet directly is to keep the spreadsheet from trying to parse the data. If you have a way of keeping it from doing that, you can open the CSV straight from the spreadsheet program.
  • Write a spreadsheet formula to append a curly open bracket, a curly closed bracket, and a comma to single line, and paste it where I need it. In Excel or in LibreOffice that formula might be, untested:
    ="{ "&A1&" },"
  • Copy the results of that formula from the spreadsheet, and paste it into the IDE.

I do this fairly often, for the same reason that it seems the OP wants to: to use handily captured or calculated data as known input to a program under development. My memory tells me that the compiler doesn't even complain if you leave the last unnecessary comma in the code. If you're a purist, you may want to take it out.

You might want to take the intermediate step of pasting the data from the spreadsheet into a text editor, copying it again, and pasting from that into the IDE. I've seen the IDE balk at big clipboards from Excel, but not from Notepad or Gedit, and those programs handle pastes from spreadsheets with aplomb.

Of course, all that presumes that you can readily get data out of a 2D PROGMEM array. I've never tried it. This post says it has a solution; maybe it works - Multi-dimensional array in PROGMEM (Arduino Noob) - Syntax & Programs - Arduino Forum. Alternatively, you could use a one-dimensional array, and index it with something like 4*i + j, where i is the row index, and j the column, zero-based. If you go that route, you don't need brackets around each line, you only need to add a comma at the end of each.

I note that the sample data shows five entries per line, rather than four, and that the sample code applies that information inconsistently. Maybe that data is for illustration, and not the real thing. If the CSV has line numbers, you'll want to account for it somehow, either by allowing space for it in the array, or stripping it out of the data. The array already takes up nearly a third of the available flash memory; I'd recommend removing line numbers. Here's a way to strip the first entry off a line of comma-separated values in Excel or Libre, untested:=MID(A1,FIND(",",A1)+1,LEN(A1))OpenOffice might do it that way, too; I don't remember.

As I see it, that solves the problem, as it was directly stated: it make the data available to the program, it doesn't require the OP to manually change oodles of lines of code, and it uses the tools he says he already has.

16

My memory tells me that the compiler doesn't even complain if you leave the last unnecessary comma in the code

No your memory is lying to you.

In these situations I have used a small processing program to read the data file and print it out with comers and braces.
Then I simply copy from the processing window and paste into the arduino IDE.

17

Quote from: mrburnette on May 15, 2014, 07:41:23 pm
Your statement that I bolded above is not helpful., IMO.
Apparently not. I thought it hinted at a solution to the OP's stated problem. Since it didn't seem to work, I'll be explicit this time.

Addressing the OP's directly stated concerns,

Very nice, detailed and helpful. Should someone do a search in the future and this thread comes up, your level of detail will be useful.

I am well aware that it is difficult when answering these queries to decide to just "link" in an effort to spark a flame or to be more detailed and write a few paragraphs. I use both. Often the explanation is simply more complex that I would wish to discuss with an Op that is already having fundamental issues, if I have example code, I'll just link to it and let cut&paste take if from there. When the process works as it should, then a member utilizing search in the future can perhaps gain enough insight that they can move forward independently. These dialogs, hopefully, will have value beyond just the immediate concern.

Ray

18

Grumpy_Mike:
No your memory is lying to you.

Not yet, I think. In a test program, this line compiles, and the array elements show up where I expect them:

int16_t a[3][4] = { {0,1,2,3}, {4,5,6,7}, {8,9,0,1}, };

Note that last unnecessary comma before the last curly bracket. No complaints from the IDE, even when I asked it to be verbose.

19

tmd3:
I thought it hinted at a solution to the OP's stated problem. Since it didn't seem to work, I'll be explicit this time.

Thank you for the much more explicit post. And thanks to Grumpy_Mike for the discussion of whether this will work. This is a very good discussion.

You're correct that my CSV has line numbers/timestamps which I was including in the sample variables but I was just going to delete the column in Excel and resave the CSV so it's 1200x4, sorry for the confusion. Also I realized that the first 3 columns won't exceed 255 and are unsigned so I can use bytes for those, so it'll end up being a 1200x3 byte and a 1200x1 int, as opposed to a 1200x4 int. That still won't fit in flash memory but it will help anyway.

Unfortunately I don't have time to play with this today but on Monday I'll try some of the things you mention of just pasting the csv in in various ways including simple excel formulas and seeing if it will take it. If I can't get it to then I'll just write a program to add the curly braces, that shouldn't be too difficult, I just didn't want to do it if it was unnecessary.

Grumpy_Mike, you don't happen to have some sample code for when you add the curly braces in processing do you?

Has anyone done something like this where they stored data with Progmem and seen if it will work? I still haven't had a chance to look and see how I will pull the data back out, I'm assuming there's a way.

I may just order an SD card shield, can anyone confirm whether that will definitely work for this? Then I don't have to worry about progmem and possibly running out of program memory. Or the suggestion for separate memory on i2c. Any pro/cons to separate memory vs sd card?

20

tmd3:
This post says it has a solution; maybe it works - Multi-dimensional array in PROGMEM (Arduino Noob) - Syntax & Programs - Arduino Forum.

That post looks very encouraging, very similar to what I'm trying to do. Also I think your excel formula approach should work really well.