Pages: [1] 2   Go Down
Author Topic: Option to include sketch in flash?  (Read 1680 times)
0 Members and 1 Guest are viewing this topic.
San Francisco Bay Area
Offline Offline
Newbie
*
Karma: 0
Posts: 8
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I've been an embedded developer for years, and have enjoyed using the Arduino for its simplicity and flexibility.

I was totally taken by surprise when a new programmer asked me why the source code isn't downloaded to the flash with the executable.  The obvious answer is that "It isn't needed.", followed by "That just isn't done."

Then I thought about it and realized the brilliance of the idea.  There is no technical reason this couldn't be offered as an option.  Most sketches are small relative to the available space on a '328 (the pde file for the Blink sketch is just 509 bytes).

The IDE could then detect a sketch on the attached board and extract it to a tmp file, allow you to edit it, then re-flash it whenever you upload.  There have been several times this would have simplified life for me, where I deployed an Arduino in a project for a while (like a year), then I wanted to make a small change, but I wasn't 100% sure what version of which sketch I had installed on it.

I realize it isn't a typical embedded development approach, but then Arduino is all about bringing embedded programming to the masses.

So, what do you think?  Would it be possible to offer a per-sketch option to store the sketch to the board along with the executable (assuming there is sufficient space), and then an option to extract a sketch from an attached Arduino for editing?

Cheers,
 - Dean
Logged

Phoenix, Arizona USA
Offline Offline
Faraday Member
**
Karma: 36
Posts: 5519
Where's the beer?
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

In theory, you could probably do this right now with PROGMEM (it wouldn't be quite so automatic, though)...
Logged

I will not respond to Arduino help PM's from random forum users; if you have such a question, start a new topic thread.

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 170
Posts: 12482
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
but I wasn't 100% sure what version of which sketch I had installed on it.
building a version string /signature in your sketch to be displayed at start up could solve that for less than 20 bytes(?)

That said, I like the idea but it should be optional as some of my sketches leave no room even without source in ti smiley
In fact the idea is quite old, every interpreting computer - e.g. the Commodore 64 -  did/does store the source in memory
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Manchester (England England)
Online Online
Brattain Member
*****
Karma: 514
Posts: 31529
Solder is electric glue
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
So, what do you think?
Well it would make the downloading process much longer and it would limit the program size, just so someone who forgot what code was downloaded into the system could get it back.
Not much of a gain for the pain.
Logged

San Francisco Bay Area
Offline Offline
Newbie
*
Karma: 0
Posts: 8
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
but I wasn't 100% sure what version of which sketch I had installed on it.
building a version string /signature in your sketch to be displayed at start up could solve that for less than 20 bytes(?)
That would be a great start, but it still requires me to maintain good source control discipline.  I have to remember to bump the version number each time, and I have to make sure all the versions that live longer than one debug cycle are in a repository.

This suggestion is more in the spirit of making the development process simpler, as opposed to improving the developer smiley-wink

That said, I like the idea but it should be optional as some of my sketches leave no room even without source in ti smiley
In fact the idea is quite old, every interpreting computer - e.g. the Commodore 64 -  did/does store the source in memory

Completely agree that it needs to be optional.  As I said "... offer a per-sketch option...".  Also, I believe the option should be ignored if the sketch won't fit.   In that case, the IDE should upload the executable as expected, and just present a warning that the sketch couldn't also fit.

Cheers,
 - Dean
« Last Edit: June 16, 2011, 05:45:32 pm by Quazar » Logged

San Francisco Bay Area
Offline Offline
Newbie
*
Karma: 0
Posts: 8
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
So, what do you think?
Well it would make the downloading process much longer and it would limit the program size, just so someone who forgot what code was downloaded into the system could get it back.
Not much of a gain for the pain.

As I said in my original post: "... offer a per-sketch option ...".  I am not suggesting this be forced on all the users.  I would personally use it all the time because my sketches tend to be simple sequencing programs which are not particularly large, but I certainly understand that it isn't for everybody.

Cheers,
 - Dean
Logged

SF Bay Area (USA)
Offline Offline
Tesla Member
***
Karma: 106
Posts: 6378
Strongly opinionated, but not official!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The idea has usefulness that is somewhat inversely proportional to how likely it is to work.
It works really well for small sketches like BLINK, but who cares?  As your sketch gets bigger and more complicated, it would be more useful, but it's less likely to work :-(

Saving a copy of the source doesn't get rid of the need for reasonable discipline in version control.  If you have 16k of source with no other markings (like version numbers), and are trying to figure out what it does compared to other source for the same sketch that is 6 months newer (or older), you're probably ... in trouble.  In fact, carefully marking up the source code (with an "edit history") is what I did back before I'd heard of version control software, and it's still a good idea.

I wonder if some intermediate level of information stored in flash would be nearly as useful?  Sketch name and date, perhaps (+ username + computername + ip address?  Gotta make those arduino chips traceable in case someone uses them for evil purposes.)
Logged

San Francisco Bay Area
Offline Offline
Newbie
*
Karma: 0
Posts: 8
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The idea has usefulness that is somewhat inversely proportional to how likely it is to work.

It works really well for small sketches like BLINK, but who cares?  As your sketch gets bigger and more complicated, it would be more useful, but it's less likely to work :-(

It would work for much more complicated sketches than just Blink.  It either works or it doesn't, and you can discover that at the time you compile your sketch so you can adjust your development style accordingly.  I think some folks would use this all the time when possible, and some folks would never use it.

Saving a copy of the source doesn't get rid of the need for reasonable discipline in version control.

Sure it does.  Obviously if there are intermediate versions that I care about, then it is up to me to archive them responsibly - this is true of any content I create.  But with Arduino, most of the time I only care about the code that is on the board right now.  In fact, I can't think of a single time that I've needed an arbitrary intermediate version of a sketch, but I can think of dozens of times that I've had to go back and make sure I picked up development from the right version of a sketch.

To provide a more concrete example:  I make haunted house animatronics, and I use lots of Arduinos to run servos, pneumatics, lights, etc.  The sketches are pretty straight forward sequencing (wait for a PIR sensor to fire, cycle through a fixed sequence and then reset).  Each prop is hand made and slightly unique.  Also, each year, the exact needs of the haunted house may be slightly different (might want to tweak the timing).

So, when I am setting up a prop that needs tweaking, I have to count on my notes to make sure I'm not loading the wrong firmware.  I'd much rather plug in my laptop (or any random laptop for that matter), extract/tweak/upload - done.  No chance that I picked the wrong firmware, and I really don't care about the timings that the haunted house 3 years ago used.

I agree that from a computer science purist perspective it is a sloppy approach, but in this real-world situation it would absolutely save me time and reduce mistakes.

If you have 16k of source with no other markings (like version numbers), and are trying to figure out what it does compared to other source for the same sketch that is 6 months newer (or older), you're probably ... in trouble.

I'm not suggesting that.  I simply suggesting putting the current source with the executable.  It isn't intended to help you compare versions, or determine history.  It is just intended to keep the source and the product together in one place.

Version control is always a good idea.  But having a copy of the source on the device that you are 100% sure matches the code on the device eliminates all variables and mitigates bad version control practices, hard drive failures, etc.  It also works if you give the project to a friend and they want to continue hacking on it.

I wonder if some intermediate level of information stored in flash would be nearly as useful?  Sketch name and date, perhaps (+ username + computername + ip address?  Gotta make those arduino chips traceable in case someone uses them for evil purposes.)

That would be fantastic, and I'd support getting that info into the flash 100% of the time.  I'd also love to see the IDE include integrated version control, even if it is just something trivial and linear.  But all that aside, I still don't see why putting the source code into flash would ever be a bad thing if it fits.

Cheers,
 - Dean
Logged

SF Bay Area (USA)
Offline Offline
Tesla Member
***
Karma: 106
Posts: 6378
Strongly opinionated, but not official!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'd hate to encourage people to make their source code smaller by, say, omitting comments.  (Those were the "bad old days"!)
Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 170
Posts: 12482
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
This suggestion is more in the spirit of making the development process simpler, as opposed to improving the developer

So what we need is some extra code that runs under the upload button of the IDE (or in the make file) that:

- add an unique identifier to the code: uint8_t UUID[] = genUUID(- http://www.ietf.org/rfc/rfc4122.txt - http://en.wikipedia.org/wiki/Universally_unique_identifier -  );
  16 bytes in binary / 32 bytes in ascii format, maybe add some tags around:
   "<UUID=12345678912345678912345678900000>" makes 39 bytes
  "<UUID>BBBBBBBBBBBBBBBB" makes 22 bytes (B = binary encoded)
  the UUID can easily be extracted from hexdump

- appends the UUID to some textfile that contains:  
  * date, time
  * sourcename, location
  * more ...

- or better writes a copy of the source to your personal cloud ..\..\myCloud\ArduinoSketches\$UUID$.pde


This approach would cost only 22 bytes of flash memory and gives the means to recapture all sources of all sketches made. The size of the sketch is not relevant as not the sketch itself is stored but a reference to it. The reciprocal usefullness (thanks westfw for this magnificent term) is replaced by a constant usefullness smiley-wink

besides the UUID info like filename date and time purpose etc could be included as ASCII but that costs substantial more bytes (overkill => don't).

Other discussion:
// should the IDE get something like javadoc ==>  ArduinoDoc?
// Javadoc is a tool for generating API documentation in HTML format from doc comments in source code.


my 2 cents
Rob
« Last Edit: June 17, 2011, 02:13:33 am by robtillaart » Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

San Francisco Bay Area
Offline Offline
Newbie
*
Karma: 0
Posts: 8
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

... or better writes a copy of the source to your personal cloud ..\..\myCloud\ArduinoSketches\$UUID$.pde

That would be a great feature, and it could be used to provide a full revision history as well.  That would probably be a more widely useful feature than what I am asking for.  I can see a few disadvantages, although I admit they are minor:
1) Cloud pollution.  Thousands of sketch versions building up over the years, and you don't know which 20 you actually need to keep.
2) Handoff.  When somebody else wants to tweak one of my creations, they may not have access to my cloud
3) No internet.  I've had to set up a few haunted houses in areas with no wifi (parking garages), though that is a shrinking problem these days.

I'm going to stick to my guns and say I still want an option to include the sketch in the flash.  Here's why:

I am a believer in simplifying things as much as practical.  If the source is in the flash, then a stranger can pick up an object I created 10 years prior and start hacking it.  There is a reason that scripting languages are so popular: the source IS the executable.  Scripting is computationally less efficient than running compiled code, and yet it is extremely popular.  This is at least in part because you don't have to worry about loosing the source and you can always tweak a script.  Saving the sketch in the flash gives you the simple development cycle of a scripted language with the computational efficiency of a compiled language.

I checked and only one of my projects is too big to fit both the executable and the sketch in 32K, and that project was complex enough that I had done proper source control on it anyway.  Dozens of my other sketches would easily fit.  So, all that unused flash is busy storing nothing but 0xff's.  I'd rather put it to use doing something potentially helpful than let it sit there unused.

Again I ask, where is the harm in an option to include the source if it fits?

Cheers,
 - Dean
« Last Edit: June 17, 2011, 12:14:14 pm by Quazar » Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 170
Posts: 12482
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Again I ask, where is the harm in an option to include the source if it fits?
There is no harm as long as it is an option which is default off.

The harm can be that
* I want have access to the source of the sketch, but don't want to publish the source to my customer. (intellectual property)
* The size of larger sketches do not fit anymore (mentioned before)
   especially well documented sketches => I have several sketches that are 25K+ in size
* one need to store the used libraries too if you want to be able to debug it in detail ... or at least there version numbers
* IDE version, compiler version etc need to be stored too (that are not so much byts)
* ....

Maybe even more, but if I must be able to recreate my PC configuration which I needed to compile and debug this app one need to go this far
(in the medical IT world companies even have working windows 3.1 compiler configs as customers still have them in the field.... That is the only way to give 100% service.
In theory - and I think quite far in practice - my proposal can realize this. It can store the compile environment including version numbers of IDE, avrtools and OS name/version (although it doesn't need to backup those)

OK, for hobby purposes it doesn't need so "extreme" but just to give you food for thought smiley-wink

Rob

 

Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

San Francisco Bay Area
Offline Offline
Newbie
*
Karma: 0
Posts: 8
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Again I ask, where is the harm in an option to include the source if it fits?
There is no harm as long as it is an option which is default off.

I have never suggested any other approach.  Strictly an opt-in behavior.

The harm can be that
* I want have access to the source of the sketch, but don't want to publish the source to my customer. (intellectual property)

Of course.  Opt-in makes this a non-issue.

* The size of larger sketches do not fit anymore (mentioned before)
   especially well documented sketches => I have several sketches that are 25K+ in size

If it doesn't fit, then it won't get written to flash and there can be no harm.
In every one of my posts, I've indicated this should be an option, and it should only happen if it can fit.  I don't see how I could be more clear on these points.

* one need to store the used libraries too if you want to be able to debug it in detail ... or at least there version numbers
* IDE version, compiler version etc need to be stored too (that are not so much byts)
* ....

Are these things in the sketch file?  That is pretty much all that conventional source control schemes handle unless you go out of your way to manage the additional info somehow.  The sketch file is all I've ever bothered to archive, and that has always been sufficient for my needs.

Maybe even more, but if I must be able to recreate my PC configuration which I needed to compile and debug this app one need to go this far
(in the medical IT world companies even have working windows 3.1 compiler configs as customers still have them in the field.... That is the only way to give 100% service.
In theory - and I think quite far in practice - my proposal can realize this. It can store the compile environment including version numbers of IDE, avrtools and OS name/version (although it doesn't need to backup those)

OK, for hobby purposes it doesn't need so "extreme" but just to give you food for thought smiley-wink

These are issues that even large corporations wrestle with:  How to reliably recreate the exact bits that were shipped for any arbitrary release.  It is not a problem that I'm attempting to solve, nor is it something that most hobbyists care deeply about.  I think just storing the source file is sufficient for the vast majority of uses, and would be an improvement over what is stored now (which is nothing but the executable).


To be clear:  I am a fan of the cloud solution - it has some very nice features and could be the basis for some pretty awesome new capabilities.  (Arduino development social network?).  I am also a fan of optionally storing the sketch file when uploading if it will fit in the leftover space.  I don't think these two capabilities need to be mutually exclusive.

Cheers,
 - Dean
« Last Edit: June 17, 2011, 02:42:19 pm by Quazar » Logged

Global Moderator
Netherlands
Offline Offline
Shannon Member
*****
Karma: 170
Posts: 12482
In theory there is no difference between theory and practice, however in practice there are many...
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Hi Dean,

Don't get me wrong, I like the concept of storing sketches in the device, it is an inspiring problem and got me thinking. It has several advantages (especially finding the right code as you pointed out) no doubt about that, but it is not a final solution.

This problem is called deployment management. [very recognizable]

As a developer I need a solution that I can trust, it should work every time I want to. Storing sketches in the Arduino is not allways possible (due to size) and therefor I cannot rely on it.

So if one wants to spend energy in solving deployment management for Arduino, we should think of a way that is:
- transparant for the programmer - (don't do things that can be automated --> KISS)
- configurable (switch on/off etc)
- works for all deployments
- robust, reliable
- and so on.

Storing a sketch in an Arduino does not work allways, as it fails on a crucial point imho SIZE. That doesn't mean it has no value, on contrary it can be very usefull as you pointed out, I am just stating it isn't reliable enough. Storing a reference to the source (etc) takes 16 bytes (UUID) which is 0.05% of the available memory and independant of sketch size. And yes there will be applications that don't even have these 16 bytes free. A real final solution should even work for this case. That means that the reference to the code should be stored in the Arduino but at the same time can't be stored in the Arduino. This is a typical TRIZ contradiction.

Solving that contradiction => the binary code itself is the reference (mmm 32K keys, no good...)

making 32K keys more practical: after uploading a sketch, AVRdude reads the complete memory back and makes a SHA1 hash to be used as reference to the sourcecode. So if one arduino comes back through the mail one can read its memory back, do the SHA1 hash and one has the reference to the sources.

That said, this reference will not be the only way to access the sources, full text indexing of all your sketches is very well possible these days, so such things need to be in the final solution too.

 The complexity to realize SketchWithin, SHA1 and UUID is comparable. The differences between the SHA1 and UUID versions are
- SHA1 will generate a new code for every source iteration, where with the UUID this is optional
- UUID uses (at least) 16 bytes, SHA1 uses 0 bytes of Arduino memory
- SHA1 will detect image tampering, UUID will not  (key lost??)
- UUID will probably be faster than SHA1.

My final choice would be using UUID, and the SHA1 at release moments. The cases I need the last 16 bytes I should really consider an new larger platform.

EPILOGUE:
In short storing a sketch in the Arduino is usefull in many cases as you pointed out. However it won't solve the "what version of code have I deployed problem" in all cases. The SHA1 and UUID solutions will perform better especially for large sketches. My choice would be using UUID all the time, and the SHA1 at release moments. The cases I need the last 16 bytes I should consider an new platform smiley

Again thanks for this inspiration,

Rob
Logged

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

"The old Europe"
Offline Offline
Edison Member
*
Karma: 1
Posts: 2005
Bootloaders suck!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Just use GIT and make regular backups to external media of your choice like everybody should do it.

It's so easy these days. 16GB USB sticks (40MB/s) cost almost nothing, external 2.5'' disks are almost free considering the storage space they offer. Unless we're talking about backing up a video collection.

If the AVR chips had 'quasi unlimited' storage, I wouldn't oppose as much though. But I would never rely on the source code being stored on the chip as well. Murphy's Law would get you anyway, trust me on that. It's much better to keep your valuable code in good condition and safe somewhere else.
Logged

• Upload doesn't work? Do a loop-back test.
• There's absolutely NO excuse for not having an ISP!
• Your AVR needs a brain surgery? Use the online FUSE calculator.
My projects: RGB LED matrix, RGB LED ring, various ATtiny gadgets...
• Microsoft is not the answer. It is the question, and the answer is NO!

Pages: [1] 2   Go Up
Jump to: