How to keep a copy of the uploaded program and be sure to find it months later?

The problem is that if I make changes to a file every few minutes and upload a new version I don't have the discipline to create a new directory and a new file every time I change my mind. Also (due to my poor discipline) I change the file after uploading it to Arduino X so that I can try something new on Arduino Y - and get myself all mixed up.

Well, in the new 1.6.5 IDE there appears to be a new "feature" that wants me to save the file when I'm not ready. I think I click the cancel button and move on. When I'm done working with the file I then use the "save as" option. I think i know better when I want to save something than does the new IDE.

so I can go back to that version and re-compile it and upload it and expect it to be identical.

Assuming you use the same compiler + compiler settings yes so you should keep all your versions of the compiler too, ans associate the code version with the right compiler.

robtillaart:

so I can go back to that version and re-compile it and upload it and expect it to be identical.

Assuming you use the same compiler + compiler settings yes so you should keep all your versions of the compiler too, ans associate the code version with the right compiler.

I agree. But I think you have quoted out of context.

I am hoping @MorganS will explain how to do that in the context of Subversion. In other words, how, exactly does Subversion help (if it does).

...R

You can't expect to go back to a version you uploaded in the middle of a programming session, although that is possible. Was the version compiled at 2:04:24 the one that did something useful or was it the one that you broke something?

But once you finish a session, before you turn the computer off, it is often useful to save a checkpoint there. Either it is the "finally got it talking to the sensor!" version or it is the "not sure why it's not talking to the sensor?" version, taking a minute to write that comment as part of a Subversion 'commit' is very valuable.

Bot Git and Subversion start with that concept of a unit of work. You have finished something. Now it is time to save that to the repository. It may be a complete working system, like you just loaded that code onto 100 Arduinos ready to ship or it may be a single feature within a system.

Looking at this in Subversion, you can review the "log" for each file. Each time it was committed, what was the comment you wrote at the time. If you have got yourself tied in knots and the sensor code you thought was working isn't, then you can go back and find that early version where it was working. You can also use the "branch and tag" method, where you can have multiple developers working on different features in branches and use tags to record major events like a version release. Branches aren't useful for a single developer and tags are just like the log except you can't write long comment descriptions.

Can you also associate more files to the project, like drawings, parts list and documentation?

@dorvakta, please don't sidetrack this Thread with queries about Subversion. If you want to follow that up please start your own Thread.

@MorganS, Thanks for your comments. What you have described is pretty much how I thought Subversion works. I specifically want to avoid the need for a "commit" action on my part.

I just want to be able to retrieve the code that created the program that happens to be on my Arduino as identified by a Serial.print() message from the Arduino - regardless of whether I uploaded it in the middle of a session or at the end of it. For example, the message might be Serial.println("TestInoC-YYYYMMDD-HHMMSS");

I have been writing some stuff which I will post tomorrow. I don't have time to finish it now.

...R

dave-in-nj: ummm.. have the IDE append either the first or last lines

// last saved, 5 jan 16 - 523 bytes, last uploaded 22dec15 - 613 bytes

thinking about this, if in setup, one printed the file name , date, size, etc. somehow you would have to grab that from the file system when you compile, you get something like : Sketch uses 450 bytes (1%) of program storage space. Maximum is 32,256 bytes. Global variables use 9 bytes (0%) of dynamic memory, leaving 2,039 bytes for local variables. Maximum is 2,048 bytes. if this data were passed to a print command, then it would display when the serial monitor is connected. Serial.print("Sketch name :"); Serial.println(" ? ? ? ? ? "); // get from the IDE Serial.print("System date:"); Serial.println(" ? ? ? ? ? "); // get from the IDE Serial.print("Verify Data:"); Serial.println(" ? ? ? ? ? "); // get from the IDE it would load the data from, would hope, only that session for the verify data. and your housekeeping would require you run verify before you upload. in that way the data is created for that session.

This Arduino system is proving very inconvenient.

I had assumed if I had a file (say myFile.h) in a directory beside the .ino file - such as

Master
    Master.ino
    lib
       myFile.h

that the Arduino system would copy all of lib into the temp directory it uses for compiling. But in fact it just ignores it. It has to be like

Master
    Master.ino
    myFile.h

Then I discovered that even though I use #include "myFIle.h" it will also draw in all the files from a library called myFile as though I had used #include <myFile.h> and that gives rise to multiple definiton errors.

I’m sure I can workaround this, but it is a real PITA and suggests that the designers of the system never considered that users might want variants of common libraries. But, of course, that is also reflected in their scant regard for backward compatibility.

Work continues as the hair gets greyer. :slight_smile:

…R

I just put these few lines in setup...

  Serial.begin(115200);
  Serial.print(F(__FILE__));                          // Always display sketch name and date info
  Serial.print("\t");
  Serial.print(F(__DATE__));
  Serial.print("\t");
  Serial.println(F(__TIME__));

And after uploading the sketch create a zip archive of the folder/sketch. That way I can still tweak the code but can always recover the correct sketch uploaded to the MCU from the name/date/time stamp of the file.

Robin2: Work continues as the hair gets greyer. :)

...R

hair...HAIR ???? you still have hair ?

Riva: I just put these few lines in setup...

And after uploading the sketch create a zip archive of the folder/sketch. That way I can still tweak the code but can always recover the correct sketch uploaded to the MCU from the name/date/time stamp of the file.

That's more or less what I am working towards - but I want it to be automatic so I don't forget - which is what my Python script does.

But I want it to also save any non-standard library files that are needed (for example the RF24 library) - which is the part that is currently leading to grey hair. I would prefer not to have to part company with the Arduino system.

...R

Robin2: That's more or less what I am working towards - but I want it to be automatic so I don't forget - which is what my Python script does.

huh? which does your Python script do? save ? or forget ?

dave-in-nj: huh? which does your Python script do? save ? or forget ?

It forgets of course. I have no problem remembering :)

Or maybe I have that backwards.

...R

Finally, some light at the end of the tunnel.

I think I now have a workable solution and I would like to describe the concept before I complete the Python code in case people have better suggestions.

I wanted, as far as possible, to create a system that is consistent with the normal Arduino way of doing things, and I think I have achieved that.

I have split this into two separate Replies so as to keep the basic concept of keeping track of versions from the more comples issue of keeping track of library files etc.


First, the basic system ...

Let's assume I have a file Master.ino and it may or may not be in a directory called Master

At the very top of that file there are a few lines of build options as C/C++ comments - like this // python-build-start // action, upload // board, arduino:avr:uno // port, /dev/ttyACM0 // ide, 1.5.6-r2 // python-build-end

My Python program uses this data to formulate a call to the IDE command line.

Before compiling the program the Python code writes a line into the .ino file like char archiveDirName[] = "Master-YYYYMMDD-HHMMSS"; and I will have written these corresponding lines in setup() Serial.print("Archived as "); Serial.println(archiveDirName);

If the program is uploaded successfully Python will make a directory (in a special Archive Directory) called Master-YYYYMMDD-HHMMSS and will put a copy of the .ino file in it.

Later, I can run the Arduino program, see "Archived as Master-YYYYMMDD-HHMMSS" in the Serial Monitor, and easily find the actual code that was used to create the program. There will be no reason for me to modify the code in the Archive Directory.

This is my basic concept for keeping track of the code that was used to generate an Arduino program.

And this system will easily (and automatically) handle the situation where a slightly different version of Master.ino is uploaded onto a different Arduino - the time stamps will be different.

As it stands, this is a single user design. If two or more people were working on a project it would be possible for Master.ino to be uploaded onto two separate Arduinos at the exact same time. I will leave it as an exercise for anyone faced with that risk to find a solution for it.

... more in the next Reply

...R

The next thrilling installment ...

I mentioned in an earlier Reply that I am also concerned to ensure that I know exactly which version of the IDE and what libraries were used to generate the program - and, at some future date, to be able to use the exact same libraries to recreate the program.

As well as libraries, my wireless application involves separate Master and Slave .ino files which use a shared .h file with some common data, such as the wireless channel and data rate.

My idea for keeping track of the libraries that are used to generate code, and the project specific .h files requires three simple elements.

First, there are the libraries that are included with the IDE when you download it. If I save a copy of the download file that contains the IDE I will always be able to reproduce that version of the IDE and all its libraries. It is a simple matter manually to put a copy of that file in the Archive Directory.

Second, there are the extra libraries (such as TMRh20's RF24 library) that I download separately. For those librarires all I need to do is save a copy of the libraries alongside the code. That is the reason for saving the archive copy of the .ino file in a directory. The Python program makes a note of any of those extra libraries that are referred to in the .ino file.

Third, with the regular Arduino IDE the only way I have found for referring to a shared .h file that is not in the same directory as the .ino file is to use the full path name

include "full/path/to/nnn.h"

and that system works with my Python script

This Python program also has the (for me) very positive side effect that it is no longer necessary to create my file Master.ino in a directory called Master. The Python program will copy the program so that the Arduino requirements are complied with. It also works if Master.ino is inside a directory called Master.

When the upload is successful, Python will copy the .ino file, all the library files and any .h files into the Archive Directory called Master-YYYYMMDD-HHMMSS. That way, if I ever need to recreate the Arduino program I can easily find the exact libraries and .ino file that were used to create the program.

It may help if I illustrate the directory structures that I have in mind.

For writing code ... TrainControl Master.ino Slave.ino lib CommonRxTx.h

For compiling Master.ino ArduinoTemp ArduinoTemp.ino (which is a copy of Master.ino)

For archiving Master.ino ArduinoUploadArchive arduino-1.5.6-r2-linux32.tgz (the file containing the Arduino IDE with its libraries) Master-YYYYMMDD-HH0722 Master.ino CommonRxTx.h libraries TMRh20RF24 TMRh20RF24.h etc Master-YYYYMMDD-HH0833 Master.ino CommonRxTx.h libraries TMRh20RF24 TMRh20RF24.h etc

Someone may say that this involves a lot of duplication of files on my hard disk, and that is quite true. But hard disk space is too cheap to be bothered by that.

All comments on either or both Replies will be very welcome.

...R

Yes, in the past this scheme would have created an unbelievable number of files on your hard drive. Now that storage is so cheap, it will be as believable as the number of photos on your phone.

I still maintain that commit comments are necessary and only required when a unit of work is finished. But if each Arduino emits the file and date time, then it is possible to find the right version among hundreds when trying to match a running Arduino to its source.

Question: If you compiled and uploaded to a board and then noticed a typo in a comment, then saved the file without uploading, does your system still keep a copy of the corrected code which was not uploaded?

MorganS: Question: If you compiled and uploaded to a board and then noticed a typo in a comment, then saved the file without uploading, does your system still keep a copy of the corrected code which was not uploaded?

No. As written it only (and always) creates an archive copy following a successful upload.

I also have the program written so that if it does not find a line in the .ino file with char archiveDirName[] it won't bother archiving the file. This allows it to work with simple programs that don't need to be archived.

I will post the Python code, perhaps on Friday next, after I have used it a bit more - in case some really stupid error presents itself :)

I agree with you about comments - I generally keep notes as I go along. And I make backup copies of programs at strategic development points.

But the problem this Python program should solve is "How do I know which is the code I used to write the program that is running on this Atmega 328 that I 'found'." And this is particularly relevant for a program for a wireless system because you cannot communicate with the wireless system without knowing exact things like the channel and ID number of the device.

...R

OK. I think it is now working the way I want it. I will post the code in a separate Reply for ease of viewing, but please use the attachment if you want to try it because I have used tabs and the Forum software loses them. Also, please change the file name from PyArduinoBuilder.txt to PyArduinoBuilder.py

Be sure to read through the code before trying it to make sure it will not do something you object to

You can run the program with python path/to/PyArduinoBuilder.py path/to/myArdinoProg.ino

I use the program with the Geany text editor. Select Build/Set Build Commands and in the bottom section Execute Commands give it a name (I use ArduinoBuild) and enter the coomand in the middle box. My command is
/mnt/sdb1/SGT-Prog/Arduino/PythonUploader/PyArduinoBuilder.py "%d/%f".
Then you can verify or upload the Arduino code by clicking Build/ArduinoBuild.

Note that you could create the command in the C++ Commands section but that does not open a terminal window which is very inconvenient when there is a Java error. Without the terminal window I have to find the process and kill it. With the terminal window I can just click Ctrl-C.

As written the Python program creates (if they don’t exist) two directories alongside itself. One ArduinoTemp is where the .ino file is copied before the IDE is called to compile it. The other ArduinoUploadArchive is where the archive copies of the Arduino code are stored.

I hope there are sufficient comments in the program to make sense of it. You need to include comments at the top of your Arduino program which the Python program reads so it knows what to do. These are described in the Python code.

Please remember that this is something I created for my own use, without any attempt at “polish”. Also, please keep in mind that it is a script that is intended to run once from top to bottom. Consequently there did not seem any reason to put the code into functions.

This code is an evolution of the system in this Thread.

As usual, comments are welcome.

…R

PyArduinoBuilder.txt (7.06 KB)

Here is the text of the Python program for ease of reference. Please use the attachment in the prvious post if you want to try the code.
Be sure to read through the code before trying it to make sure it will not do something you object to

#!/usr/bin/env python

# simple program to compile and upload Arduino code using the Arduino command line

print 
print "====PyArduinoBuilder====="
print

#===========================

import subprocess
import sys
import os
import shutil
import time
import fileinput

#===========================
	#	create some background data
	#		these need to reflect the details of your system

	#	where is the Arduino program
arduinoIdeVersion = {}
arduinoIdeVersion["1.5.6-r2"] = "/mnt/sda4/Programs/arduino-1.5.6-r2/arduino"
arduinoIdeVersion["1.6.3"] = "/mnt/sda4/Programs/arduino-1.6.3/arduino"

	#	where are libraries stored (/mnt/sdb1/SGT-Prog/Arduino/ is my Sketchbook directory)
arduinoExtraLibraries = "/mnt/sdb1/SGT-Prog/Arduino/libraries"

	#	where this program will store stuff
	#		these directories will be beside this Python program
compileDirName = "ArduinoTemp"
archiveDirName = "ArduinoUploadArchive"

	#	default build options
buildOptions = {}
buildOptions["action"] = "verify"
buildOptions["board"] = "arduino:avr:uno"
buildOptions["port"] = "/dev/ttyACM0"
buildOptions["ide"] = "1.5.6-r2"

	#	some other important variables - just here for easy reference
compileDir = ""
archiveDir = ""
arduinoProg = ""
inoFileName = ""
inoBaseName = ""
inoArchiveDirName = ""
archiveRequired = False
usedLibs = []
hFiles = []

#============================
	#	ensure directories exist
	#	and empty the compile directory

	#	first the directory used for compiling
pythonDir = os.path.dirname(os.path.realpath(__file__))
compileDir = os.path.join(pythonDir, compileDirName)
if not os.path.exists(compileDir):
	os.makedirs(compileDir)

existingFiles = os.listdir(compileDir)
for f in existingFiles:
	os.remove(os.path.join(compileDir,f))

	#	then the directory where the Archives are saved
archiveDir = os.path.join(pythonDir, archiveDirName)
if not os.path.exists(archiveDir):
	os.makedirs(archiveDir)

#=============================
	#	get the .ino file and figure out the build options
	#
	#	the stuff in the .ino file will have this format
	#	and will start at the first line in the file
	#		// python-build-start
	#		// action, verify
	#		// board, arduino:avr:uno
	#		// port, /dev/ttyACM0
	#		// ide, 1.5.6-r2
	#		// python-build-end

inoFileName = sys.argv[1]
inoBaseName, inoExt = os.path.splitext(os.path.basename(inoFileName))

numLines = 1  # in case there is no end-line
maxLines = 6
buildError = ""
if inoExt.strip() == ".ino":
	codeFile = open(inoFileName, 'r')

	startLine = codeFile.readline()[3:].strip()
	if startLine == "python-build-start":
		nextLine = codeFile.readline()[3:].strip()
		while nextLine != "python-build-end":
			buildCmd = nextLine.split(',')
			if len(buildCmd) > 1:
				buildOptions[buildCmd[0].strip()] = buildCmd[1].strip()
			numLines += 1
			if numLines >= maxLines:
				buildError = "No end line"
				break
			nextLine = codeFile.readline()[3:].strip()
	else:
		buildError = "No start line"
else:
	buildError = "Not a .ino file"

if len(buildError) > 0:
	print "Sorry, can't process file - %s" %(buildError)
	sys.exit()

	#	print buid Options
print "BUILD OPTIONS"
for n,m in buildOptions.iteritems():
	print "%s  %s" %(n, m)
print

#=============================
	#	get the program filename for the selected IDE
arduinoProg = arduinoIdeVersion[buildOptions["ide"]]

#=============================
	#	prepare archive stuff
	#
	#	create name of directory to save the code = name-yyyymmdd-hhmmss
	#	this will go inside the directory archiveDir
inoArchiveDirName = inoBaseName + time.strftime("-%Y%m%d-%H:%M:%S")
	#	note this directory will only be created if there is a successful upload
	#	the name is figured out here to be written into the .ino file so it can be printed by the Arduino code
	#	it will appear as char archiveDirName[] = "nnnnn";

	#	if the .ino file does not have a line with char archiveDirName[] then it will be assumed
	#		that no archiving is required
	#	check for existence of line
for line in fileinput.input(inoFileName):
	if "char archiveDirName[]" in line:
		archiveRequired = True
		break
fileinput.close()

if archiveRequired == True:
	for line in fileinput.input(inoFileName, inplace = 1):
		if "char archiveDirName[]" in line:
			print 'char archiveDirName[] = "%s";' %(inoArchiveDirName)
		else:
			print line.rstrip()
	fileinput.close()
	#~ os.utime(inoFileName, None)

#=============================
	#	figure out what libraries and .h files are used
	#	if there are .h files they will need to be copied to ArduinoTemp

	#	first get the list of all the extra libraries that exist
extraLibList = os.listdir(arduinoExtraLibraries)

	#	go through the .ino file to get any lines with #include
includeLines = []
for line in fileinput.input(inoFileName):
	if "#include" in line:
		includeLines.append(line.strip())
fileinput.close()
print "#INCLUDE LINES" 
print includeLines
print

	#	now look for lines with < signifying libraries
for n in includeLines:
	angleLine = n.split('<')
	if len(angleLine) > 1:
		libName = angleLine[1].split('>')
		libName = libName[0].split('.')
		libName = libName[0].strip()
			#	add the name to usedLibs if it is in the extraLibList
		if libName in extraLibList:
			usedLibs.append(libName)
print "LIBS TO BE ARCHIVED"
print usedLibs
print

	#	then look for lines with " signifiying a reference to a .h file
	#	NB the name will be a full path name
for n in includeLines:
	quoteLine = n.split('"')
	if len(quoteLine) > 1:
		hName = quoteLine[1].split('"')
		hName = hName[0].strip()
			#	add the name to hFiles 
		hFiles.append(hName)
print ".h FILES TO BE ARCHIVED"
print hFiles
print

#==============================
	#	copy the .ino file to the directory compileDir and change its name to match the directory
saveFile = os.path.join(compileDir, compileDirName + ".ino")
shutil.copy(inoFileName, saveFile)

#===============================
	#	generate the Arduino command
arduinoCommand = "%s --%s --board %s --port %s %s" %(arduinoProg, buildOptions["action"], buildOptions["board"] , buildOptions["port"], saveFile)
print "ARDUINO COMMAND"
print arduinoCommand

#===============================
	#	call the IDE
print "STARTING ARDUINO -- %s\n" %(buildOptions["action"])

presult = subprocess.call(arduinoCommand, shell=True)

if presult != 0:
	print "\nARDUINO FAILED - result code = %s \n" %(presult)
	sys.exit()
else:
	print "\nARDUINO SUCCESSFUL"
		#	if we were not uploading that is the end of things
	if buildOptions["action"] != "upload":
		sys.exit()

#================================
	#	after a successful upload we may need to archive the code
if archiveRequired == True:
	print "\nARCHIVING"
		#	create the Archive directory
	arDir = os.path.join(archiveDir, inoArchiveDirName)
	print arDir
		#	this ought to be a unique name - hence no need to check for duplicates
	os.makedirs(arDir)
		#	copy the code into the new directory
	shutil.copy(inoFileName, arDir)
		#	copy the .h files to the new directory
	for n in hFiles:
		shutil.copy(n, arDir)
		#	copy the used libraries to the new directory
	for n in usedLibs:
		libName = os.path.join(arduinoExtraLibraries, n)
		destDir = os.path.join(arDir, "libraries", n)
		shutil.copytree(libName, destDir)
	print "\nARCHIVING DONE"

sys.exit()

#==============================

Enjoy …

…R