Pages: [1] 2 3   Go Down
Author Topic: "External" memory for program memory?  (Read 3680 times)
0 Members and 1 Guest are viewing this topic.
Worcester, Ma.
Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I know, from what I have read, that the stock Arduino device comes with a fixed amount of memory that can be used for running sketches, ranging from about 4k, to 256k or so, depending on the Atmel chip used.

I also know that by using a SD shield, or an Ethernet shield that has a built-in SD card reader, I can use the SD chip for data-logging and such.

Question #1:
Has anyone tried to use the SD card's memory for extended data storage, used programmatically?  For example:  I want to use an Arduino with an Ethernet shield, (containing an SD card reader), as a web server and I want to put multiple web-pages on the, (much larger!), SD card rather than main memory?

Question #2:
Is it possible to use the external memory on the SD card as additional storage for larger sketches than main memory will hold?  Can I execute out of the external memory, or is there a paging scheme that would work?

If this hasn't been done, than this is an area ripe for research, but if it HAS been done - at least in part - I'd really rather not re-invent the wheel again if I don't have to.

Thanks!
Logged

Global Moderator
UK
Offline Offline
Brattain Member
*****
Karma: 302
Posts: 26249
I don't think you connected the grounds, Dave.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I don't imagine this question hasn't been asked before - did you search the forum for previous answers?
Logged

"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

Valencia, Spain
Offline Offline
Faraday Member
**
Karma: 146
Posts: 5469
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Question #1:
Has anyone tried to use the SD card's memory for extended data storage, used programmatically?  For example:  I want to use an Arduino with an Ethernet shield, (containing an SD card reader), as a web server and I want to put multiple web-pages on the, (much larger!), SD card rather than main memory?

I have.

Isn't that what SD cards are for...?

Question #2:
Is it possible to use the external memory on the SD card as additional storage for larger sketches than main memory will hold?  Can I execute out of the external memory, or is there a paging scheme that would work?

If this hasn't been done, than this is an area ripe for research, but if it HAS been done - at least in part - I'd really rather not re-invent the wheel again if I don't have to.

ATmega Chips can write to their internal flash, yes. The Arduino bootloader does it all the time. You could load sketches off the card with a similar scheme.


Logged

No, I don't answer questions sent in private messages (but I do accept thank-you notes...)

Worcester, Ma.
Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I don't imagine this question hasn't been asked before - did you search the forum for previous answers?

Absolutely!  And I even varied the search terms too.

Unfortunately, even the best forum software often is sadly lacking in search capabilities.

Thank you for your suggestion, but rest assured that I did all the "RTFM!!" steps prior to asking my stupid and silly questions.
Logged

Worcester, Ma.
Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Question #1:
Has anyone tried to use the SD card's memory for extended data storage, used programmatically?  For example:  I want to use an Arduino with an Ethernet shield, (containing an SD card reader), as a web server and I want to put multiple web-pages on the, (much larger!), SD card rather than main memory?

I have.

Isn't that what SD cards are for...?

Oh so true. . . .

However, if I had $20 for every time there was a silly restriction on how some memory media can be used, I'd be a gazillionare today.  And it would not surprise me if there was some restriction within the Atmel chip that allows the bootloader to load from USB, but won't allow similar activity to happen at run-time.

Sorry to have asked such a silly question, and I do thank you for the time you took to answer it.
Logged

Poole, Dorset, UK
Offline Offline
Edison Member
*
Karma: 52
Posts: 2278
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

In reply to question 2 yes you can it called overlay programming. Its a very old trick from the 1950's/1960's but ..... well look it up.

Mark
Logged

Worcester, Ma.
Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

In reply to question 2 yes you can it called overlay programming. Its a very old trick from the 1950's/1960's but ..... well look it up.

Mark

Mark,

You are absolutely correct and it has gone under a variety of names over the years.  In fact, modern operating systems use a virtual memory scheme that - at the kernel level - is rather similar.

I was thinking about trying something like that, but wanted to ask first BEFORE banging my head against the floor trying to do something that everyone else knew could not be done.

Thanks for the update!
Logged

Poole, Dorset, UK
Offline Offline
Edison Member
*
Karma: 52
Posts: 2278
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You may also like to look at code relocation where addresses are resolved at load time or using  only addressing via registers for both jumps and load/store op's. But to be realistic you need to asm only! and it's very hard work to write such programs. Better to get a bigger CPU!.

Mark
Logged

Phoenix, Arizona USA
Offline Offline
Faraday Member
**
Karma: 40
Posts: 5572
Where's the beer?
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

If you wanted to approach this idea from another angle, one way to do what you want to do would be to write an interpreter that would fit within the limits of the SRAM and Flash; there is likely more than enough space available for that.

It would no longer be an "Arduino" (that is, you couldn't develop using the normal Arduino tools) but that may not be that big of an issue. You'd have to make the interpreter very lean, and it would likely only run on a single ATMega processor. The biggest issue would likely be how to keep the SD card reader code tight and lightweight. It would take a lot of work, but it should be possible. It wouldn't be as fast as a regular Arduino, given the interpretation overhead, but it would work well for most tasks, I think. It might even be easier to develop for.

That's basically what the Parallax Basic Stamp 2 was - a PIC16F57 microcontroller with an external 24LC16B EEPROM. The PIC ran at 20 MHz, with 2K (12 bit) words of flash, 72 (8 bit) bytes of SRAM; the EEPROM was a 16K I2C device. There shouldn't be any reason why you couldn't do the same with the 328 and a similar EEPROM or an SD card.
Logged

I will not respond to Arduino help PM's from random forum users; if you have such a question, start a new topic thread.

Global Moderator
UK
Offline Offline
Brattain Member
*****
Karma: 302
Posts: 26249
I don't think you connected the grounds, Dave.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I think the OOPIC took the same approach - interpeted code in external EEPROM.
Logged

"Pete, it's a fool looks for logic in the chambers of the human heart." Ulysses Everett McGill.
Do not send technical questions via personal messaging - they will be ignored.

Austin, TX
Offline Offline
Full Member
***
Karma: 2
Posts: 182
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hate to rain on your parade, but I don't think this is possible (I mean putting sketch instructions on an SD card and having the arduino execute them).  At the very least, it's going to be insanely complicated.

First of all, the VM paging and program overlay methods you mention are for computers using a Von Neumann architecture.  These computers have the instructions and data in the same memory space.  Therefore they can read instructions into memory and then set the program counter to that memory location to start executing them.
The arduino is a Harvard architecture in which the instructions and data are in separate memory spaces.  It cannot execute an instruction stored in the RAM.  So you can't read a compiled sketch from an SD card into RAM and then start executing it.
So how does the arduino put a sketch into program memory? The bootloader sucks up a block of instructions from the serial port into RAM and then uses some special instructions to stuff it into program memory.  There are lots of limitations: you have to do a block (page) at a time, the timing is critical, the source and dest addresses have to be aligned correctly, etc. See the data sheet for more information.  The bootloader and the program are segregated so that you can't trash your bootloader by downloading a sketch.

It may be possible to write a sketch that reads instructions from an SD card into RAM and then loads them into the program memory.  I doubt it would be easy, though. You can't over-write any part of the code that is doing the reading including the SD interface.  Since you can't be sure where the sketch is in program memory, you'd probably have to write the sketch in assembler, like the boot loader.
Plus there's the problem of knowing WHEN to read in a page of instructions. There are no page faults on an arduino. By the time you write the code (probably in assembly) for the SD interface, pushing into progmem, and doing the code paging, you've probably eaten up most of your memory anyway.  It would be much simpler to just use an arduino with more memory.
Then there is the problem of writing the code that goes on the SD card.  You wouldn't be able to use regular arduino sketches, that's for sure.

It would be an enormous coup to get it to work in some sort of reasonably usable fashion.  I think it would be really slow, though, and the code would be fragile.  Probably hard to use on different arduino boards, too.

Dealing with the memory limitations (both RAM and progmem) of the arduino is just part of the experience.  If you want more than that, you need to look at a different kind of microcontroller.  There are atmel chips that support external RAM.  And things like Raspberry Pi that have full VM support.
Logged

Chris J. Kiick
Robot builder and all around geek.

Global Moderator
Boston area, metrowest
Online Online
Brattain Member
*****
Karma: 534
Posts: 26970
Author of "Arduino for Teens". Available for Design & Build services. Now with Unlimited Eagle board sizes!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

What about Bitlash?

http://bitlash.net/bitlash-users-guide.pdf

BITLASH INTERPRETER
1. Runs in under 16kB on the AVR ‘328
2. Parses Bitlash code on the fly from RAM, PROGMEM, EEPROM, or SD card
3. Executes commands live from the command line and runs background tasks
Logged

Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

Offline Offline
Edison Member
*
Karma: 33
Posts: 1438
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

PicAxes actually have commands that can do this, but then they use bytecodes passed to the internal BASIC interpreter. Of course, PicAxes or so limited in memory, that they *have* to be able to do this. 8^)
Logged

Offline Offline
Full Member
***
Karma: 0
Posts: 115
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I know, from what I have read, that the stock Arduino device comes with a fixed amount of memory that can be used for running sketches, ranging from about 4k, to 256k or so, depending on the Atmel chip used.

I also know that by using a SD shield, or an Ethernet shield that has a built-in SD card reader, I can use the SD chip for data-logging and such.

Question #1:
Has anyone tried to use the SD card's memory for extended data storage, used programmatically?  For example:  I want to use an Arduino with an Ethernet shield, (containing an SD card reader), as a web server and I want to put multiple web-pages on the, (much larger!), SD card rather than main memory?

Question #2:
Is it possible to use the external memory on the SD card as additional storage for larger sketches than main memory will hold?  Can I execute out of the external memory, or is there a paging scheme that would work?

If this hasn't been done, than this is an area ripe for research, but if it HAS been done - at least in part - I'd really rather not re-invent the wheel again if I don't have to.

Thanks!


1 yes it's pretty trivial to do.  It does use a pile of sram as the sd card is fat formatted thus block oriented so your reading 512b at a clip. Please be specific I'm assuming your talking about flash as main memory though that's what most would call sram.  If not take a look at progmem http://www.arduino.cc/en/Reference/PROGMEM before thinking you need to deal with sdcards.

2 Yes but it's a great big PITA far easier to get a chip with more flash like the 1284 than to fiddle around with this.  This method is also very SLOW and wastes sram to do so.  If you think you need more flash than a 1284 provides consider if an 8 bit micro is the right processor type for your application.   
Logged

0
Offline Offline
God Member
*****
Karma: 39
Posts: 988
Get Bitlash: http://bitlash.net
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You beat me to it, CrossRoads, thanks.

Bitlash 2.0 (http://bitlash.net) can run script from SD card.  Your application can be as big as you like.

The sample sketch that does this can be found here: https://github.com/billroy/bitlash/blob/master/examples/bitlashsd/bitlashsd.pde


-br
« Last Edit: October 09, 2012, 05:24:31 pm by billroy » Logged

Pages: [1] 2 3   Go Up
Jump to: