very long array to arduino?

Projects Interfacing w/ Software on the Computer
1

Hi
I need to send a very long array which is stored in an excel file with each field as in :

{254,32,254,16,0,117,24,179},
{254,64,254,16,0,118,24,179},
{254,128,254,16,0,119,24,179},
{191,4,254,8,0,122,24,179},
...

I am already loading it 576 lines at a time but it means reloading every 1 hour! How can I have the Mega read this directly from excel or a txt file every time it finishes the 576 lines which was loaded in an array as in:

int TarhQuant = 576;
const uint16_t Tarh [][8] PROGMEM =

So now I need the mega when it reaches the end of the loop after 576 iterations to reload automatically new 576 lines of data?
As it can be seen I am forced to use PROGMEM and maybe if the loading can be done live then smaller chunks could be loaded avoiding PROGMEM.

thanks for your help

2

How can I have the Mega read this directly from excel

Unless Excel is sending the data to the serial port, you can't.

So now I need the mega when it reaches the end of the loop after 576 iterations to reload automatically new 576 lines of data?

You can't write to PROGMEM at run time, so the requirement and the code snippet don't go together.

As it can be seen I am forced to use PROGMEM

No, I can't see that. I can see that you have used the PROGMEM keyword in the snippet, but you have not explained why. Typically, you would read a record and process that data. Repeat until all 576 records have been read and processed.

3

I would expect the data coming from Excel to be loaded in an array in SRAM for fast access and perhaps sketch manipulation. Why put it in PROGMEM or EEPROM unless you need to save it thru a power loss? Both have a limited number of write operations available to them (lifetime).
If you have data that you need to keep thru power losses, then perhaps an FRAM with 10 to 100 trillion write cycle capability would be better.
These are all 5V, SPI interfaced parts
http://www.digikey.com/product-search/en/integrated-circuits-ics/memory/2556980?k=fram&k=&pkeyword=fram&pv1989=0&FV=ffecc2d8%2C268001d%2Cfff40027%2Cfff80434%2C450000f%2C142c002e&mnonly=0&newproducts=0&ColumnSort=0&page=1&stock=1&quantity=0&ptm=0&fid=0&pageSize=25
A SMT to DIP adapter can make it easy to use
Narrow part
http://www.proto-advantage.com/store/product_info.php?products_id=2200001
Wide part
http://www.proto-advantage.com/store/product_info.php?products_id=2200002

4

Thanks all for prompt and helpful responses.
1 - Now I know PROGMEM has limited life.
It is not necessary to worry about power loss etc. If the data is stored in excel then it can be retrieved if necessary.
2 - I need to know if I keep an spreadsheet or text data file open and when the mega determines it needs new data can it then access that data from the computer that the mega is connect to?
thanks

5

when the mega determines it needs new data can it then access that data from the computer that the mega is connect to?

No. What it CAN do is send a message to the application on the other end of the serial port. If that application understands the message, it can send back a reply. But, it all depends on the application on the PC to send the data. The Arduino can not just go get it.

6

Thanks now I understand. So it cannot open an external file and retrieve data from it but it can signal an application that it needs data and then the other app must send the data to serial and mega can receive it. This is fine but one last question how big a chunk of data that could be maximum?

7

BTW as soon as I get a chance I will use an SD card reader connected to the Mega but this was for a very quick fix that maybe I could implement by software temporarily.

8

This is fine but one last question how big a chunk of data that could be maximum?

That depends on how fast you read the data. The Serial buffer is 64 bytes. If you read faster than it fills, you can send 1000's of bytes. If not, you can't send more than 64 bytes.

9

So with SRAM being 8 KBytes or 8192 bytes, I can theoretically bring in via serial maximum 128 chunks of 64 byte data lots and store them in an array and when needed to repeat the process?

10

Yes.
Or use a '1284P based board, 16K bytes SRAM.
I offer one with SD card, battery backed RTC (DS1307), onboard or off board FTDI adapter, screw terminal option also.
http://www.crossroadsfencing.com/BobuinoRev17/

1045×620 115 KB

1047×813 153 KB

11

I can theoretically bring in via serial maximum 128 chunks of 64 byte data lots and store them in an array and when needed to repeat the process?

No. The Serial object's (there are 4 of them) input and output buffers occupy SRAM, too, as do all string literals, global variables, and the stack, all of which reduce the amount of free memory that can be used to store the "chunks of data".

Why can't you deal with one record at a time? Why do you need to store multiple records?

12

This '1284P based board, 16K bytes SRAM. will be a good choice for the next step when I would have a little more time to implement it.

thanks

13

PaulS:
No. The Serial object's (there are 4 of them) input and output buffers occupy SRAM, too, as do all string literals, global variables, and the stack, all of which reduce the amount of free memory that can be used to store the "chunks of data".

Why can't you deal with one record at a time? Why do you need to store multiple records?

right now since using the FLASH memory I have 7,440 bytes free for for local variables which I intend to use and each line of my data being 8 bytes it might cover the 576 lines needed. The code is for a kind of textile machine and going back and forth across the serial port might be very slow, although I need to test it and see if it can request and retrieve data in under 50 mili seconds or less then it would be fine until I connect and SD card soon.

14

If you upped the data rate to 115,200, then 576 bytes x 10 bits/byte x 1 second/115200 bits = 50mS.

"The Serial buffer is 64 bytes." That is a setting in the Serial.h or Serial.cpp file or some similar file that can be increased a lot if needed.

15

"The Serial buffer is 64 bytes." That is a setting in the Serial.h or Serial.cpp file or some similar file that can be increased a lot if needed.

As long as you realize that the value is the size of the incoming and outgoing buffer for each serial instance. On the Mega, doubling the value from 64 to 128 results in 1024 bytes of memory being allocated to the buffers. As long as you have lots of memory, you can do that.

Reading the data so the buffer doesn't fill up is almost always a better idea.