Show Posts
Pages: [1]
1  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 18, 2011, 09:46:26 pm
Don't get me wrong, I like the concept of storing sketches in the device, it is an inspiring problem and got me thinking. It has several advantages (especially finding the right code as you pointed out) no doubt about that, but it is not a final solution.

This problem is called deployment management. [very recognizable]

I've also heard it variously called Release Control, Release Engineering, Product Engineering, and other various things over the years.  I have yet to find a company that has completely nailed it.

As a developer I need a solution that I can trust, it should work every time I want to. Storing sketches in the Arduino is not allways possible (due to size) and therefor I cannot rely on it.

I find this an interesting view, because the '328 (or any processor) has a finite set of resources that your program could eventually outgrow.  There are always "grey area" resources like circular logs in RAM or extra debug ports that you jettison along the way as resources get tight.  Source-in-flash is just another such thing that you could use until the program outgrows it.  You then decide that (1) it is OK to stop using this feature, or (2) it is valuable to your workflow and you find a bigger chip.

Embedded developers are always making tradeoffs about how to deploy the resources of the platform.  Would you give up on circular log buffers or debugging ports because some designs don't have enough resources to afford them?  Of course not - they are tools that you deploy when appropriate.  Source-in-flash is just another tool to be deployed when appropriate.  That doesn't make it unreliable, it just makes it another decision in the tradeoff calculations.

I do get your point that just storing a reference to the source is orders of magnitude less expensive than storing the source, drastically altering the tradeoff decision.

So if one wants to spend energy in solving deployment management for Arduino, we should think of a way that is:
- transparant for the programmer - (don't do things that can be automated --> KISS)
- configurable (switch on/off etc)
- works for all deployments
- robust, reliable
- and so on.

Storing a sketch in an Arduino does not work allways, as it fails on a crucial point imho SIZE. That doesn't mean it has no value, on contrary it can be very usefull as you pointed out, I am just stating it isn't reliable enough. Storing a reference to the source (etc) takes 16 bytes (UUID) which is 0.05% of the available memory and independant of sketch size. And yes there will be applications that don't even have these 16 bytes free. A real final solution should even work for this case. That means that the reference to the code should be stored in the Arduino but at the same time can't be stored in the Arduino. This is a typical TRIZ contradiction.

Solving that contradiction => the binary code itself is the reference (mmm 32K keys, no good...)

making 32K keys more practical: after uploading a sketch, AVRdude reads the complete memory back and makes a SHA1 hash to be used as reference to the sourcecode. So if one arduino comes back through the mail one can read its memory back, do the SHA1 hash and one has the reference to the sources.

That said, this reference will not be the only way to access the sources, full text indexing of all your sketches is very well possible these days, so such things need to be in the final solution too.

 The complexity to realize SketchWithin, SHA1 and UUID is comparable. The differences between the SHA1 and UUID versions are
- SHA1 will generate a new code for every source iteration, where with the UUID this is optional
- UUID uses (at least) 16 bytes, SHA1 uses 0 bytes of Arduino memory
- SHA1 will detect image tampering, UUID will not  (key lost??)
- UUID will probably be faster than SHA1.

My final choice would be using UUID, and the SHA1 at release moments. The cases I need the last 16 bytes I should really consider an new larger platform.

In short storing a sketch in the Arduino is usefull in many cases as you pointed out. However it won't solve the "what version of code have I deployed problem" in all cases. The SHA1 and UUID solutions will perform better especially for large sketches. My choice would be using UUID all the time, and the SHA1 at release moments. The cases I need the last 16 bytes I should consider an new platform smiley

Again thanks for this inspiration,

This reasoning all seems sound to me.  I also agree that if you are down to the last 16 bytes and can't afford space for the UUID, you are probably ready to look for another platform.  I think the utility of your solution would be high enough that I'd go find a way to shrink the image by 16 bytes rather than forgo the UUID.

As for the SHA1, I agree with tastewar that if it is a hash of just the executable image, then it won't capture things like updated comments, formatting changes, or even logic changes that compile down to the same opcode stream.  If it is always used in conjunction with (and includes in its hash) the UUID, then it would cover trivial source changes.  Also, I think you would need to know the range of memory to calculate hash - you wouldn't want to do the full 32K since that may include garbage from prior uploads, and even runtime flash data.

 - Dean
2  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 18, 2011, 09:18:54 pm
Just use GIT and make regular backups to external media of your choice like everybody should do it.

It's so easy these days. 16GB USB sticks (40MB/s) cost almost nothing, external 2.5'' disks are almost free considering the storage space they offer. Unless we're talking about backing up a video collection.

If the AVR chips had 'quasi unlimited' storage, I wouldn't oppose as much though. But I would never rely on the source code being stored on the chip as well. Murphy's Law would get you anyway, trust me on that. It's much better to keep your valuable code in good condition and safe somewhere else.

I do use source control, and I do backup to external media, and I even make periodic off-site backups.  I still have to keep careful notes to ensure I've got the right version of a project for a given board, and I still get it wrong sometimes because I forgot to update my notes when I did a quick last-minute rev on a sketch in the field.  Loosing source code is not the problem I'm trying to solve, I'm trying to come up with a fool-proof way of associating a specific source version with the physical object it goes with.

I sometimes velcro a USB stick to larger things like test equipment to keep track of relevant source/drivers/scripts/manuals/notes.  This works extremely well, except when somebody borrows it and looses the USB stick (in which case I make a new USB stick from my backups).  I was just proposing a built-in version of this for Arduino.

 - Dean
3  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 17, 2011, 02:20:19 pm
Again I ask, where is the harm in an option to include the source if it fits?
There is no harm as long as it is an option which is default off.

I have never suggested any other approach.  Strictly an opt-in behavior.

The harm can be that
* I want have access to the source of the sketch, but don't want to publish the source to my customer. (intellectual property)

Of course.  Opt-in makes this a non-issue.

* The size of larger sketches do not fit anymore (mentioned before)
   especially well documented sketches => I have several sketches that are 25K+ in size

If it doesn't fit, then it won't get written to flash and there can be no harm.
In every one of my posts, I've indicated this should be an option, and it should only happen if it can fit.  I don't see how I could be more clear on these points.

* one need to store the used libraries too if you want to be able to debug it in detail ... or at least there version numbers
* IDE version, compiler version etc need to be stored too (that are not so much byts)
* ....

Are these things in the sketch file?  That is pretty much all that conventional source control schemes handle unless you go out of your way to manage the additional info somehow.  The sketch file is all I've ever bothered to archive, and that has always been sufficient for my needs.

Maybe even more, but if I must be able to recreate my PC configuration which I needed to compile and debug this app one need to go this far
(in the medical IT world companies even have working windows 3.1 compiler configs as customers still have them in the field.... That is the only way to give 100% service.
In theory - and I think quite far in practice - my proposal can realize this. It can store the compile environment including version numbers of IDE, avrtools and OS name/version (although it doesn't need to backup those)

OK, for hobby purposes it doesn't need so "extreme" but just to give you food for thought smiley-wink

These are issues that even large corporations wrestle with:  How to reliably recreate the exact bits that were shipped for any arbitrary release.  It is not a problem that I'm attempting to solve, nor is it something that most hobbyists care deeply about.  I think just storing the source file is sufficient for the vast majority of uses, and would be an improvement over what is stored now (which is nothing but the executable).

To be clear:  I am a fan of the cloud solution - it has some very nice features and could be the basis for some pretty awesome new capabilities.  (Arduino development social network?).  I am also a fan of optionally storing the sketch file when uploading if it will fit in the leftover space.  I don't think these two capabilities need to be mutually exclusive.

 - Dean
4  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 17, 2011, 12:03:46 pm
... or better writes a copy of the source to your personal cloud ..\..\myCloud\ArduinoSketches\$UUID$.pde

That would be a great feature, and it could be used to provide a full revision history as well.  That would probably be a more widely useful feature than what I am asking for.  I can see a few disadvantages, although I admit they are minor:
1) Cloud pollution.  Thousands of sketch versions building up over the years, and you don't know which 20 you actually need to keep.
2) Handoff.  When somebody else wants to tweak one of my creations, they may not have access to my cloud
3) No internet.  I've had to set up a few haunted houses in areas with no wifi (parking garages), though that is a shrinking problem these days.

I'm going to stick to my guns and say I still want an option to include the sketch in the flash.  Here's why:

I am a believer in simplifying things as much as practical.  If the source is in the flash, then a stranger can pick up an object I created 10 years prior and start hacking it.  There is a reason that scripting languages are so popular: the source IS the executable.  Scripting is computationally less efficient than running compiled code, and yet it is extremely popular.  This is at least in part because you don't have to worry about loosing the source and you can always tweak a script.  Saving the sketch in the flash gives you the simple development cycle of a scripted language with the computational efficiency of a compiled language.

I checked and only one of my projects is too big to fit both the executable and the sketch in 32K, and that project was complex enough that I had done proper source control on it anyway.  Dozens of my other sketches would easily fit.  So, all that unused flash is busy storing nothing but 0xff's.  I'd rather put it to use doing something potentially helpful than let it sit there unused.

Again I ask, where is the harm in an option to include the source if it fits?

 - Dean
5  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 16, 2011, 07:24:58 pm
The idea has usefulness that is somewhat inversely proportional to how likely it is to work.

It works really well for small sketches like BLINK, but who cares?  As your sketch gets bigger and more complicated, it would be more useful, but it's less likely to work :-(

It would work for much more complicated sketches than just Blink.  It either works or it doesn't, and you can discover that at the time you compile your sketch so you can adjust your development style accordingly.  I think some folks would use this all the time when possible, and some folks would never use it.

Saving a copy of the source doesn't get rid of the need for reasonable discipline in version control.

Sure it does.  Obviously if there are intermediate versions that I care about, then it is up to me to archive them responsibly - this is true of any content I create.  But with Arduino, most of the time I only care about the code that is on the board right now.  In fact, I can't think of a single time that I've needed an arbitrary intermediate version of a sketch, but I can think of dozens of times that I've had to go back and make sure I picked up development from the right version of a sketch.

To provide a more concrete example:  I make haunted house animatronics, and I use lots of Arduinos to run servos, pneumatics, lights, etc.  The sketches are pretty straight forward sequencing (wait for a PIR sensor to fire, cycle through a fixed sequence and then reset).  Each prop is hand made and slightly unique.  Also, each year, the exact needs of the haunted house may be slightly different (might want to tweak the timing).

So, when I am setting up a prop that needs tweaking, I have to count on my notes to make sure I'm not loading the wrong firmware.  I'd much rather plug in my laptop (or any random laptop for that matter), extract/tweak/upload - done.  No chance that I picked the wrong firmware, and I really don't care about the timings that the haunted house 3 years ago used.

I agree that from a computer science purist perspective it is a sloppy approach, but in this real-world situation it would absolutely save me time and reduce mistakes.

If you have 16k of source with no other markings (like version numbers), and are trying to figure out what it does compared to other source for the same sketch that is 6 months newer (or older), you're probably ... in trouble.

I'm not suggesting that.  I simply suggesting putting the current source with the executable.  It isn't intended to help you compare versions, or determine history.  It is just intended to keep the source and the product together in one place.

Version control is always a good idea.  But having a copy of the source on the device that you are 100% sure matches the code on the device eliminates all variables and mitigates bad version control practices, hard drive failures, etc.  It also works if you give the project to a friend and they want to continue hacking on it.

I wonder if some intermediate level of information stored in flash would be nearly as useful?  Sketch name and date, perhaps (+ username + computername + ip address?  Gotta make those arduino chips traceable in case someone uses them for evil purposes.)

That would be fantastic, and I'd support getting that info into the flash 100% of the time.  I'd also love to see the IDE include integrated version control, even if it is just something trivial and linear.  But all that aside, I still don't see why putting the source code into flash would ever be a bad thing if it fits.

 - Dean
6  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 16, 2011, 05:44:47 pm
So, what do you think?
Well it would make the downloading process much longer and it would limit the program size, just so someone who forgot what code was downloaded into the system could get it back.
Not much of a gain for the pain.

As I said in my original post: "... offer a per-sketch option ...".  I am not suggesting this be forced on all the users.  I would personally use it all the time because my sketches tend to be simple sequencing programs which are not particularly large, but I certainly understand that it isn't for everybody.

 - Dean
7  Development / Suggestions for the Arduino Project / Re: Option to include sketch in flash? on: June 16, 2011, 05:42:09 pm
but I wasn't 100% sure what version of which sketch I had installed on it.
building a version string /signature in your sketch to be displayed at start up could solve that for less than 20 bytes(?)
That would be a great start, but it still requires me to maintain good source control discipline.  I have to remember to bump the version number each time, and I have to make sure all the versions that live longer than one debug cycle are in a repository.

This suggestion is more in the spirit of making the development process simpler, as opposed to improving the developer smiley-wink

That said, I like the idea but it should be optional as some of my sketches leave no room even without source in ti smiley
In fact the idea is quite old, every interpreting computer - e.g. the Commodore 64 -  did/does store the source in memory

Completely agree that it needs to be optional.  As I said "... offer a per-sketch option...".  Also, I believe the option should be ignored if the sketch won't fit.   In that case, the IDE should upload the executable as expected, and just present a warning that the sketch couldn't also fit.

 - Dean
8  Development / Suggestions for the Arduino Project / Option to include sketch in flash? on: June 16, 2011, 03:39:20 pm
I've been an embedded developer for years, and have enjoyed using the Arduino for its simplicity and flexibility.

I was totally taken by surprise when a new programmer asked me why the source code isn't downloaded to the flash with the executable.  The obvious answer is that "It isn't needed.", followed by "That just isn't done."

Then I thought about it and realized the brilliance of the idea.  There is no technical reason this couldn't be offered as an option.  Most sketches are small relative to the available space on a '328 (the pde file for the Blink sketch is just 509 bytes).

The IDE could then detect a sketch on the attached board and extract it to a tmp file, allow you to edit it, then re-flash it whenever you upload.  There have been several times this would have simplified life for me, where I deployed an Arduino in a project for a while (like a year), then I wanted to make a small change, but I wasn't 100% sure what version of which sketch I had installed on it.

I realize it isn't a typical embedded development approach, but then Arduino is all about bringing embedded programming to the masses.

So, what do you think?  Would it be possible to offer a per-sketch option to store the sketch to the board along with the executable (assuming there is sufficient space), and then an option to extract a sketch from an attached Arduino for editing?

 - Dean
Pages: [1]