Programming External EEPROM

Suppose I have a large amount of data that I need to store in Non-volatile memory. I want to be able to upload new sketches and have that data persist. Right now I am including in my sketch, and accessing it when needed. However, as I develop my program, that data pool is growing, and is quickly going to surpass the memory limits of my device.

If I had some external non-volatile memory (would rather have EEPROM or FLASH chip, not SD card), would I be able to program that directly somehow?

Here's what I envision as a finished product: I have a sketch that is setup to read the data in 8 byte chunks, process it, and read the next 8 bytes, until it gets to the end. it does not need to save the data as it goes, so the 8 bytes would be overwrote each time it reads from the data.

I want to be able to upload new data directly to the memory, and then when the sketch runs again, it uses that new data. I will be designing a custom PCB for this, so I could have a connection for the SPI (or i2c, whatever the interface), and hook that second Arduino up to reprogram the data pool.

My issue is that my data could grow to over 200KB, so I can't just upload the whole thing to the second Arduino, and have it write to the EEPROM.

If there is anyone left reading this long winded post.....my question is, what would be my best option (outside of just reading from SD card) to make this work. Are there programs out there that I can program EEPROM right from my PC? I'm sure if can be done through a sketch on the second board and sending the data via serial, but that seems messy.

There are (or were) dedicated eeprom programmers.

Why don't you let your sketch also cater for writing data to eeprom?

I can make the sketch write to eeprom, but I would need to transfer that data in chunks somehow. If I have 100K of data, I can't just upload the whole thing in a sketch, as it would be much to large.

I'm sure it can be done with serial, but I'm not sure that would work real well. then again I'm no expert.

You can use any proper terminal program that supports files and Xon/Xoff (RealTerm, Putty). Xon/Xoff is a software handshake that can instruct the sender to wait or continue transmission.

http://forum.arduino.cc/index.php?topic=397318.msg2735345#msg2735345.

Note that
1)
This was written for internal eeprom, so you need to adjust.
2)
I can’t remember exactly why I did the things as I did them; use it as a guideline.

sterretje:
You can use any proper terminal program that supports files and Xon/Xoff (RealTerm, Putty). Xon/Xoff is a software handshake that can instruct the sender to wait or continue transmission.

http://forum.arduino.cc/index.php?topic=397318.msg2735345#msg2735345.

Note that
1)
This was written for internal eeprom, so you need to adjust.
2)
I can't remember exactly why I did the things as I did them; use it as a guideline.

I skimmed through that thread quick and I think that may be a good solution. I will have to dig into it and try it out. I can definitely modify it for external EEPROM, I haven't done it, but if I can get the data to my Arduino in chunks, I know I can get it to the EEPROM, or whatever memory I decide to use.

Thank, I will report back with what I come up with!