Arduino Forum

Forum 2005-2010 (read only) => Hardware => Interfacing => Topic started by: bhagman on Dec 15, 2009, 09:34 pm

Title: Load sketch code from SD Card
Post by: bhagman on Dec 15, 2009, 09:34 pm

If anyone is interested, I've written a bootloader which, besides interfacing normally with the IDE, also loads code from an SD Card.  So now you can update code without being connected to the IDE.

It checks a pin on reset, and if it's low, then the bootloader pulls a hex file from the SD card and programs the flash.

b
Title: Re: Load sketch code from SD Card
Post by: Osgeld on Dec 15, 2009, 09:36 pm
While I may not have a direct need for this right now, that is pretty darn spiffy
Title: Re: Load sketch code from SD Card
Post by: tweakd on Dec 15, 2009, 10:00 pm
Interesting! cant wait to see some useful ideas on what can be done with this.

What was your initial inspiration for a reason to come up with this?
Title: Re: Load sketch code from SD Card
Post by: mowcius on Dec 15, 2009, 10:35 pm
Whoa!

I wanted to do this some time ago but I don't honestly have enough programming knowledge to do it.

Please can you provide more sauce?  ;)

This will be infinitely useful.

How are you writing the hex file to the SD card?

Mowcius
Title: Re: Load sketch code from SD Card
Post by: mowcius on Dec 15, 2009, 10:37 pm
I can see my dream of programming an arduino from an arduino working soon...

Just need to get a hex compiler working in another sketch now...

Could another arduino read from the sd card and send it via serial to the other arduino?  :-/

Then it'd work!  ;)
Title: Re: Load sketch code from SD Card
Post by: ArduinoM on Dec 15, 2009, 11:55 pm
The von Neuman machines are comming!!!
RUN FOR YOUR LIVES  :o

They are MULTIPLYING !!  
Now where did I put my nanobot-powder  ::)

David.
Title: Re: Load sketch code from SD Card
Post by: florinc on Dec 16, 2009, 03:06 am
I am interested too.
How much flash memory is left for the actual application (sketch)?
Can you choose what to load or it loads a default hex file?
Thanks.
Title: Re: Load sketch code from SD Card
Post by: bhagman on Dec 16, 2009, 05:06 am
The bootloader takes up less than 4K total.  This means that you'll have 28K at your disposal (on the '328).  I'm actually using the bootloader on one of our AVR boards - with an ATmega644P - which means I have 60K left for goodies.

Right now, you can only load a hex file named "sketch.hex".  The file is in native hex format which is parsed inside the bootloader - so no fiddly changes - just copy onto the card and go.

Tw34kd:  I needed to load code on a remote installation (maybe that's obvious).  Specifically, they are for POP (point of purchase) units that play music and capture metrics.

I'll put together a page with the bootloader and other info in the next couple of days.


b
Title: Re: Load sketch code from SD Card
Post by: willich on Dec 16, 2009, 09:11 am
Yiehha, i am really interested.
Sounds damn cool. I read about all the questions and halftries in this Forum, and gave up hope that somebody enough skilled programmer has the need for this :)
Title: Re: Load sketch code from SD Card
Post by: BroHogan on Dec 18, 2009, 05:56 pm
Waiting excitedly for this! Wanted this ability for a long time.

I guess I'll have to get over my fear of burning the 644P bootloader with a parallel programmer.

Can the hex file live on the card with other files?
(Guess I can wait till Christmas to find out.)

Thanks again for your great work. This for sure is a chick magnet! You should score mightily.
Title: Re: Load sketch code from SD Card
Post by: frank26080115 on Dec 18, 2009, 06:37 pm
I think we are all interested in seeing your work
Are you using any of the popular FAT libraries?
Does it handle verification? How does it handle verification (page by page or all at once at the end)? What does it do when there is a mismatch?
Title: Re: Load sketch code from SD Card
Post by: mowcius on Dec 18, 2009, 07:58 pm
Quote
This for sure is a chick magnet! You should score mightily.

Maybe if you are looking really geeky chicks ;D

Mowcius
Title: Re: Load sketch code from SD Card
Post by: BroHogan on Dec 18, 2009, 09:28 pm
Quote
Maybe if you are looking really geeky chicks

Right! And I can certainly appreciate your firm grasp on the obvious. The comment was made with regard to bhagman's tag line.
Title: Re: Load sketch code from SD Card
Post by: mowcius on Dec 18, 2009, 09:31 pm
Quote
Right! And I can certainly appreciate your firm grasp on the obvious.

Life is obvious, i have a firm grasp on life, oh wait, maybe not  ;D

Mowcius
Title: Re: Load sketch code from SD Card
Post by: duc23 on Jan 06, 2010, 01:17 am
ping....    for any more information regarding this boot loader.

Really interested in learning more.
Title: Re: Load sketch code from SD Card
Post by: kiltjim on Jan 06, 2010, 03:44 am
Awesome!  I was looking for something like this back when I was playing with the Basic Stamp.  I had Palm, and would have loved to be able to program the device out in the field.  There wasn't a  straight forward method to doing it though.  Of course, at that time there wasn't anything like an XBee.

Now is there any way to edit a .pde file from a smart phone?
Title: Re: Load sketch code from SD Card
Post by: PaulS on Jan 06, 2010, 01:54 pm
You might be able to edit a .pde file on your smart phone, but how would you compile it? You need to write the resulting .hex file to the SD card, to be loaded by the Arduino, not the edited .pde file.
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 06, 2010, 03:06 pm
That's what I was looking into some time ago. I was looking at an arduino doing the compiling and then programming another arduino.

I didn't get very far becuase I realised that that was rather out of my reach with limited knowledge (none) of the bootloader/chip architecture/arduino IDE etc.

I got as far as that you could make sketches to run on arduino but you would need something on the arduino to interpret it (bitlash or similar). Then you could change the code and run different programs at different times, create them on anything as long as you could save it as a text file on a microSD card (or any media that can be interfaced with an arduino). With the new arduino chips (on the mega etc) with a lot more space, bitlash (or similar) could be made so that it could run most commands that you would ever want to run on an arduino.

In theory, you could make your own program that could interpret C++ code and run it using a program on your arduino. Then you wouldn't need to compile it as a hex (or whatever the chip uses), you could have your arduino interpret it as it runs it. It would not be as fast as straight forward 'code' but for most things, that would not matter too much.

Anyone want to embark on a C++ interpreter version of bitlash with me?
It is a project on my list but I have been a bit busy recently and not had much chance to start on this (possibly massive) project.

Mowcius
Title: Re: Load sketch code from SD Card
Post by: RanTalbott on Jan 06, 2010, 04:31 pm
Quote
You might be able to edit a .pde file on your smart phone, but how would you compile it?


Just get a Linux-based smartphone,  and install arduino-0017 on it   ;)
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 06, 2010, 06:43 pm
Quote
Just get a Linux-based smartphone,  and install arduino-0017 on it

Hehe, maybe another project for the arduino team, a mobile friendly version (if there could be such a thing!)

Mowcius
Title: Re: Load sketch code from SD Card
Post by: RanTalbott on Jan 06, 2010, 06:43 pm
Quote
Anyone want to embark on a C++ interpreter version of bitlash with me?

Nobody that you'd trust unsupervised with sharp objects  8-)

A C++ (or even C) interpreter or compiler is completely impossible:  it just doesn't have the needed memory resources to load or run one,  even slowly.

If you want to run an interpreter on an Arduino,  you need to scale back your expectations to a language like FORTH or Tiny BASIC.  Or something like bitlash.

Ran
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 06, 2010, 08:20 pm
Quote
A C++ (or even C) interpreter or compiler is completely impossible:  it just doesn't have the needed memory resources to load or run one,  even slowly.

If you want to run an interpreter on an Arduino,  you need to scale back your expectations to a language like FORTH or Tiny BASIC.  Or something like bitlash.

Ok, I thought it would be difficult...

Now I think about it, I see what you mean. It is not just simple commands, you are asking it to interpret curly brackets, variables and everything in between.

I will look more into bitlash. I think I saw another attempt at an interpreter type thingy somewhere? Anyone know what it was?

Mowcius
Title: Re: Load sketch code from SD Card
Post by: bhagman on Jan 10, 2010, 08:58 pm
FYI: I'm still working on documenting it better, but here's the code and a short write-up.  I'll make up some fancy connection diagrams and such.

http://www.roguerobotics.com/wikidocs/code/bootloader_using_ummc

b
Title: Re: Load sketch code from SD Card
Post by: lgeek on Jan 11, 2010, 08:09 am
Regarding the interpreter you were talking about earlier, I was thinking about that a couple of days ago and I belive that it is perfectly doable if the script is compiled to some form of bytecode before being copied to the storage medium.
It would be slow even if "interpreting" compiled code that could run natively on the board, but I'm positive it could work. Probably this sounds weird, but:
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 11, 2010, 01:17 pm
Quote
I'm still working on documenting it better, but here's the code and a short write-up.  I'll make up some fancy connection diagrams and such.

http://www.roguerobotics.com/wikidocs/code/bootloader_using_ummc


Can it work with any SD card storage? I do not have the roguerobotics board (I am thinking of getting the rMP3). Is there something specific that is needed on the rr board?

How are you getting your hex files for putting on the microSD card, are you taking them from the file folder that the arduino IDE creates?

Mowcius
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 11, 2010, 01:20 pm
Quote
The AVR being a Harvard architecture, you can't just load the code into RAM and then jump to it (this is how the von Neumann architecture works because it doesn't differentiate between program storage and data storage).


Is there any way that we could have a Von Neumann arduino? What chips use the Von Neumann architecture?

Some ARM processors can do that can they not? I presume the ARM arduino that I saw a bit ago will be using a harvard ARM chip...

Mowcius
Title: Re: Load sketch code from SD Card
Post by: lgeek on Jan 11, 2010, 01:50 pm
Microcontrollers are built using the Harvard architecture to keep the complexity, cost and power consumption low. In contrast, I think that all modern CPU families use the Von Neumann architecture. This includes ARM and X86. It's pretty clear why if you want to run different programs or to implement a real operating system.

You could take a look at some ARM dev boards if you need something more powerful than the AVRs, but now I think I got totally offtopic.  :)
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 11, 2010, 01:52 pm
Yes, a bit off topic now. There is also a new command line for arduino floating around that could be very good with a bit of development:
http://tp://sites.google.com/site/anotherarduinobot/cmd

Mowcius
Title: Re: Load sketch code from SD Card
Post by: lgeek on Jan 11, 2010, 01:55 pm
Yeah, but it just executes different functions included in the sketch when it recognises a command.
Title: Re: Load sketch code from SD Card
Post by: mowcius on Jan 11, 2010, 02:09 pm
Yes I know but still could be made to load commands off of an SD card to then run simple sketches on the arduino...

Mowcius
Title: Re: Load sketch code from SD Card
Post by: lgeek on Jan 11, 2010, 03:04 pm
Oh, I see what you mean. The issue here would be the fact that the flash memory is guaranteed only for a relatively small number of writes. But it is doable if all the code can be squeezed in the bootloader section.
Title: Re: Load sketch code from SD Card
Post by: bhagman on Jan 11, 2010, 05:58 pm
mowcius: the Rogue Robotics board completely simplifies talking to SD cards.  It only takes about 2K of flash space in the bootloader to read from the SD card.  The bootloader will work on any of the modules (i.e. uMMC/uMP3/rMP3 - as of now).

I like the idea of a command interpreter.  I think I will work on this concept and implement a scripting engine for reading off the SD cards.  Just remember that you will be sacrificing a LOT of speed/efficiency for flexibility.  But this is often the case with bytecode interpreters.  Would be fun to make a Basic Stamp interpreter. ;)  I'm already intrigued.

b
Title: Re: Load sketch code from SD Card
Post by: duc23 on Mar 06, 2010, 05:35 pm
Resurrecting this thread as I haven't seen any solutions yet using a basic arduino and SD reader.  

This feature would be a huge benefit and open up many more possible ideas for arduino based products.

Allowing for easy field/customer updates to embedded solutions greatly increases the usefulness and flexibility of the total package.

Any further ideas on this subject?

Title: Re: Load sketch code from SD Card
Post by: mowcius on Mar 06, 2010, 05:41 pm
Quote
Any further ideas on this subject?

Yeah.

I have got an rMP3 board from roguerobitics that allows you to load sketches off an SD card with it. I have not got a programmer yet to reprogram my bootloader to try it though.  :(

I would still like to be able to read the .hex sketch off the SD card and send it via serial to program a second arduino but I lack the knowledge to do that. When I have more time I will try and have another chat with bhagman to work out how to do it.

It should also be possible with any SD cards and reading system as the hex is just text and could be interpreted with the arduino. You would need the arduino to send the 'start programming' commands to the slave arduino first and then send the finish ones at the end. Anyone know what those commands are? Anywhere I can look for it?

Mowcius
Title: Re: Load sketch code from SD Card
Post by: florinc on Mar 06, 2010, 09:00 pm
Quote
You would need the arduino to send the 'start programming' commands to the slave arduino first and then send the finish ones at the end


The "start programming" is essentially a reset. Force a reset on the target Arduino; immediately after, it enters the "wait for sketch to be uploaded" state. The source Arduino then starts sending the bytes (compiled sketch). After transmission is over, the target will start executing the freshly received sketch.
(Both Arduinos, source and target have the standard bootloader.)
Title: Re: Load sketch code from SD Card
Post by: keeper63 on Mar 07, 2010, 09:02 am
Quote
Any further ideas on this subject?


What I have yet to see (and this isn't so much Arduino as AVR) is why code like this, plus the other developments in SD storage, haven't made their way into "bootable SD cartridges" for the UzeBox.

It's had an SD card interface forever, but last I looked there weren't any interface software/code for it. The ability to throw your game .hex file on an SD card (plus its tile graphics, music, and sfx files) would allow you to distribute custom games much easier than what is done most often now.

Maybe it doesn't have as big of following as I expected?

As far as an interpretor for the Arduino is concerned, here is where I think things could shine:

Build an interpretor (and layout gui?) for a ladder logic system.

This would be well within the capabilities (well, for most ladder logic i/o modules) of the Arduino, and it would be useful in the case of bringing industrial controls (and programming) to the hobbyist level. Such a system could be used to implement custom manufacturing and assembly line controls at a smaller (and cheaper) scale, for instance in home or cottage manufacturing...?

In theory, you could even build a programming suite into the Arduino as well, using a 2x16 or 2x20 LCD (plus a keypad or other button pad) for display and input (depending on how much memory a ladder logic interpretor takes, I suppose).
Title: Re: Load sketch code from SD Card
Post by: duc23 on Mar 07, 2010, 06:26 pm
Agreed - this is a gap that needs to be filled.

What I am looking for from the outside seems simple enough.
1) Method to trigger the load of the hex file, or decide to run the already loaded program.  This could be a pin held high/low or the existence of a file on the SD card.  

2) Program existing in the bootloader which can communicate via SPI to a SD card, and load the hex file into program storage.  

Reset and we have a new sketch running...  

Since I don't have a clue to what is really involved, perhaps those much more knowledgeable could chime in and detail what parts of this we are all missing and have kept this from coming to life as of yet.

Title: Re: Load sketch code from SD Card
Post by: mowcius on Mar 08, 2010, 09:49 am
Quote
What I am looking for from the outside seems simple enough.
1) Method to trigger the load of the hex file, or decide to run the already loaded program.  This could be a pin held high/low or the existence of a file on the SD card.  

2) Program existing in the bootloader which can communicate via SPI to a SD card, and load the hex file into program storage.  

Reset and we have a new sketch running...  

Since I don't have a clue to what is really involved, perhaps those much more knowledgeable could chime in and detail what parts of this we are all missing and have kept this from coming to life as of yet.


The boards from Rogue Robotics can do this:
http://www.roguerobotics.com
The info is somewhere on there but the site is running very slowly here at the moment.

Mowcius
Title: Re: Load sketch code from SD Card
Post by: bhagman on Mar 26, 2010, 09:26 am
cr0sh: One of the reasons why you don't see much in the way of bootloader support for SD cards on the Atmel parts is because of the amount of code that needs to be crammed into less than 4K (and that's only available on the ATmega328 - the 168, can only have a 2K bootloader).  It is possible, but the result would be pretty limited.

rbrockman: the bootloader I wrote does exactly what you're looking for.  The example I've done:

http://www.roguerobotics.com/wikidocs/code/bootloader_using_ummc

will load a sketch from the SD card module (uMMC/uMP3/rMP3) if pin 4 is held low during reset.  Once done, it starts the sketch (if pin 4 has gone high).


b