Libraries on SD card to save space

Projects Programming
1

I couldn't find my question anywhere, which prompted me to ask it here. Bear with me, this is my first post!

Not long ago I ordered a 3,2" TFT LCD for a project I'm working on. The LCD comes with an SD card slot, which is a nice feature.
However libraries, and especially the LCD libraries, eat up a lot of precious memory, and since my Uno has only 32kB there is not much of it to begin with.

Is it possible to store the libraries my program uses on the SD card and have the Arduino read it from there? That would probably cut the memory my program will use in half, and give me room to put in other features.

Thanks in advance!

2

Is it possible to store the libraries my program uses on the SD card and have the Arduino read it from there?

Even if the library files were read from the SD card they would still need to be compiled into the final program so you would not save any memory.

3

Ah, shucks. I guess that's off the table then.

Thanks for the quick answer!

4

The answer was fast but incorrect. You could save compiled parts of the program to the SD. It is no problem. But I believe (not know for sure!) ATMega executes instructions from its own Flash memory only. You need either load the part of program from SD to the Flash (you could swap parts of program) or somehow force the ALU execute instructions from SD. The first possibility will be slow and ruin the Flash quickly. The second possibility would be even slower and would probably use much of program space to implement. Generaly speaking you can reasonably use the SD as large extension of inbuilt EEPROM, not Flash or RAM.

5

Ah, so it cán be done - it's just not at all practical. Thanks, I might do something with this someday for laughs, but for this project it's not what I'm looking for.

I guess I better get a Mega then :stuck_out_tongue:

6

Smajdalf:
You could save compiled parts of the program to the SD. It is no problem.

Can you please provide details of how to do this.

But I believe (not know for sure!) ATMega executes instructions from its own Flash memory only. You need either load the part of program from SD to the Flash

Again, how do you do this ?

or somehow force the ALU execute instructions from SD.

And once again, how ?

I would be interested in seeing evidence that any of your suggestions is feasible.

7

MisterMD:
I couldn't find my question anywhere, which prompted me to ask it here. Bear with me, this is my first post!

If it can be done, it would have been done a long time ago. That's why you could not find anything on it.
There are alternatives now that will give you more program space and RAM, usually ARM based boards. Get a board using ARM processor instead of a Mega.

8

UKHeliBob:
Can you please provide details of how to do this.
Again, how do you do this ?
And once again, how ?
I would be interested in seeing evidence that any of your suggestions is feasible.

ROFL, read XMega datasheet.

  1. Program is just data. You can store it anywhere, even on paper with a pen. It will be hard for processor to access it but with some OCR why not?
  2. Ever heard about self-programming? If not look it up in the datasheet. It would be strange and probably impractical bud you can store many variants of your program on the SD card and if needed rewrite the currently used by a new one. Flash may be rewriten at least 10.000 times which gives you the possibility to update twice a day for a few years for example. If you are lover of ATMega and want to use the chip for your multi-purpose device this may be quite viable strategy to quickly change behavior of the device.
  3. You could write a program to load instructions from the SD card, decide what instruction it is and execute it. I guess it would be possible to use AVR native instructions but it would be quite ineffective. My idea would be something like (in pseudo-code) this:
while (true) {
  load next instruction to X;
  case X:
     0x0000: break;
     0x0001: do instruction 0x0001; break;
     ...
     0xffff: do instruction 0xffff;
}

You can find meaning of AVR instructions on Atmel's web. This straithforward and naive way wouldn't work, you need to implements jumps correctly relative to SD code, not ATMegas's code. There would also be a mess with registers and 2 word instructions but I guess it could be done anyway if someone cared.
Another problem is speed - loading 1 word (2 bytes) instruction from SD card takes at least 32 CPU cycles so the execution would be quite slow - something like 0.5 MHz on Arduino at best. Getting better MCU looks like more effective in most applications I can think about.
But on the other hand - is there any modern DIP MCU? All I have seen (but I didn't look hard) is SMD with 100+ pins which is difficult to use for DYI.

9

@Smajdalf You have so many things wrong.

  1. If you have forgotten that the code needs to be linked. Object files need to be linked and the address of data and functions resolved before you can execute the code and in order to do this we need to know where the code is to go in memory.

  2. Only code running in the boot section of flash can load code to the rest of flash. The amount of code in the boot section is extremely limited. Try reading the datasheet.

@OP you can't doit.

Mark

10

@Smajdalf - it's not really feasible. Self-programming to which you refer is generally done by systems using the Von Neumann architecture however the Arduino uses the Harvard architecture.

Putting aside the linking issues (which would not be trivial) you would need to be able to rewrite parts of program memory by reading from the SD card. The OP does not mention that his code uses the SD card (yet), so the saving made by putting part of the code onto the SD card could be negated by needing more code to actually do this swapping. The swapping code could, of course, itself not be swapped.

Then there is the fact that reprogramming the flash takes time. There would be a big performance hit as you rewrite it. And on top of that there is the limit to the number of times you can reprogram it.

It would be far more feasible to just get a bigger chip, like the Atmega644 or Atmega1280. For a tiny amount of extra expense you get a lot more flash memory.

11

Well I do hate cannot be done. So if he asks if it is possible I try to find ways how it is possible instead of stating "you can't do it". I know there aren't straightforward solutions and are quite ineffective to implement - using another MCU would be much better. But my impression is that for "real programming" all 8-bit is dead. So you should answer any Arduino question "buy a 32-bit MCU, you get more for the same price".

So let's say someone wants to do the mentioned thing. You tell them to buy ATMega wih more flash. Is there a 2GB ATMega? I doubt it. So even if you get largest ATMega on the world you may run out of flash much sooner than you run out of "SD extended" memory. Not mentioning possibility of swaping SD cards.

It looks like noone here ever done something like this (for good reasons). So there is no such command as "progmem" for saving RAM space. Doing such thing would be quite pointless and difficult but it can be done. If someone thinks it may be worth their time it shouln't be so extremly hard anyway.

@holmes4: Why do you think that reflashing from SD card should take (much) more code than reflashing from anywhere else? Reading from SD card is easy: you just ask for data and it streams them via SPI.

@Interpreting code in SD card: there is ~100 commands in ATMega. If you manage to interpret one in 50 words the interpreting part of code will take ~10kB of flash which is not so bad for extending flash to 2 GB. But you must somehow force the linker to keep a few registers for you to use. I have no idea if it could be done easily. Speed will be major problem but if you can afford to wait for some long part of program why not?

12

Perhaps an easier approach might be to see if your code works with the Teensy 3.2 microcontroller. They have a plug-in that allows you to use the Arduino IDE for program development, but generating code for the Teensy processor. The upside is that it has 256K of Flash and 64K of SRAM and is clocked at 72MHz. Most Arduino code works with it, although I have a graphics library that I cannot get to work with it, so it's not 100% compatible. The specs are about halfway down this page:

https://www.pjrc.com/teensy/teensy31.html

13

econjack:
Perhaps an easier approach might be to see if your code works with the Teensy 3.2 microcontroller. They have a plug-in that allows you to use the Arduino IDE for program development, but generating code for the Teensy processor. The upside is that it has 256K of Flash and 64K of SRAM and is clocked at 72MHz. Most Arduino code works with it, although I have a graphics library that I cannot get to work with it, so it's not 100% compatible. The specs are about halfway down this page:

Teensy 3.2 & 3.1: New Features

Sure. 8-bit sucks, 32/64-bit FTW. From cost&time effectivity point of view the whole Arduino project is long dead useless crap. Isn't Arduino about learning (low-level) programming?

14

Smajdalf:
But my impression is that for "real programming" all 8-bit is dead.

Well, it "died" in around 1980 when people stopped using the 6502 and 6800 processors in favour of more powerful Intel chips. You know, ones that had 16 bit addresses and much more RAM. Strange, though, that 8-bit processors seem to be manufactured in their billions these days, isn't it?

See AVR microcontrollers - Wikipedia

The AVR 8-bit microcontroller architecture was introduced in 1997. By 2003, Atmel had shipped 500 million AVR flash microcontrollers.

Hardly, dead, eh?

... real programming ...

Depends what you mean by "real programming". Sure, you won't write the latest 3D shoot-em-up game on it. But these chips are easy to learn, easy to program, don't require massive operating systems in support (often with proprietary code), and are perfect for tasks like controlling toasters, ovens, doors, toy cars etc. They are small, light, and can run on tiny amounts of power. In power-down mode, waiting for an interrupt (like pressing a button on a switch) they can consume 100 nA of current.

MisterMD:
Not long ago I ordered a 3,2" TFT LCD for a project I'm working on. ...
However libraries, and especially the LCD libraries, eat up a lot of precious memory, and since my Uno has only 32kB there is not much of it to begin with.

I'm having a little trouble believing the problem exists anyway. I have a library that outputs to an LCD screen using I2C. Compiled on a Uno, it uses 5750 bytes of Flash (17%), out out 32 KB. That leaves plenty for the "real work" of whatever-it-is the OP wants to do.