I've made a modification to Optiboot bootloader, so now application can write to flash if this version of bootloader is installed.
Modified bootloader with precompiled hex files is available at GitHub - majekw/optiboot at supermaster
Installation is exactly the same as in original Optiboot.
Example is also provided (flash_program).
I hope you enjoy this long waited feature
Of course, report of success or failure would be nice I tested it on Atmega8, Atmega168 and Atmega328 without problems, but more testing could be good before it could be merged into next official release of Optiboot.
Yes, it's possible but very difficult if you want to reprogram it itself. But I know one project with its own bootloader capable to reprogram whole itself (including bootloader), so it's not impossible
The main goal of writing flash by application is to store in non-volatile memory more data than in small eeprom. Sometimes you need only few kB of such storage and connecting external flash or sd-card is overkill.
For example I use it in my project on Atmega168/328, where application code is about 8KB and the rest of flash is used for data which application uses and can modify from time to time.
Of course, you must keep in mind that flash has it's limitations as less write/erase cycles or writing only in pages, but it's still good storage for more data not frequently written.
And it's dangerous for application if you make mistake in alligning write data, but it's also easier to find than out-of-the-blue jump to bootloader code destroing random part of flash
For banks of data. Okay. But it is scary :o I don't know which flash page is available.
I would like a full functioning diagnostics webpage on my Arduino Mega with Ethernet shield if the SD fails. That webpage changes when more things are added. I would like to update that page with the html file on my computer. So you granted my wish via the bootloader I don't know if I can do it, it seems complicated
You don't need to know any adresses - just allocate some memory for array containing your static html page using PROGMEM directive with additional attribute (( aligned(SPM_PAGESIZE) )) and make it large enough to allow future updates. And it's size needs to be multiple of SPM_PAGESIZE.
It is a Mega 2560 board (SPM_PAGESIZE=256, EEPROM only 4kbyte).
The html file is larger than sram memory is available. So the flash programming could be done in chunks of 256 bytes. I think that is no problem, just increment the pointer with 256 for every chunk.
The flash_buffer may not be larger than 32kbyte. It seems to be a limit of the compiler. That’s no problem, about 16kbyte is would be okay for me.
I don’t know how to upload a file in chunks of 256 bytes, without serial handshake.
The flash programming is an awesome and unique feature, no doubt about that ! (but it is hard for me to implement).
Peter_n:
I don't know how to upload a file in chunks of 256 bytes, without serial handshake.
Split it into chunks before uploading (by hand or some javascript) and put into application only simple http endpoint which gets chunk number and 256B of data. I think it's much easier than dealing with it on chip side.
Hmmm, uploading a file with javascript in chunks of 256byte, and disabling interrupts while flash programming. It should be possible. I don't know if I have to the time to do a test.
@Peter_n, i have bad news for you, Optiboot supports chips with at most 128KB flash, so it wont work on 2560
I didn't spotted it earlier...
I think that probably it could be adapted to work on 2560 without rewriting everything, but it would be still limited by old protocol to writing only to first 128KB of flash (bootloader itself, not my modification).
Tested on a Arduino Uno SMD, and working very well
A few notes:
It is not possible to keep the contents after uploading the sketch (because the compiler creates the buffer in the flash and defines the address)
For a page size of 128:
page erase: 1.1ms
page fill: 372 us (the whole loop)
page write: 120us
That is 12us per byte or 80kbyte per second.
It is allowed to turn off the interrupts with "noInterrupt()" and "interrupt()", thus no special precautions are needed for the rest of the sketch.
Recording analog data with 1kHz sample rate, and 20kbyte flash_buffer can record 20 seconds. When it is a loop recorder, it will reach the maximum allowed write/erase cycles after 189 days. That is assuming that the data is equally spread over 160 pages.
A good way to fill it, is for example a raw file on a SD card with audio or other sampled data.
It's possible to keep contents after uploading new sketch, but not using predefined const variable like in the example.
It's possible to use any address between sketch code (aligned to page size) and bootloader. It requires some voodoo casting pointers, but in my opinion it's also safer for application (think about common 'off by one' errors and buffer placed between interrupt vectors and code, as in code generated by example).
Yes, functions in optiboot.h take care of disabling and enabling interrupts as needed in critical sections, so application could use it without special effort.
Could you try the next sketch. It is still quick and dirty, but it seems to work.
The sketch could be used with a small Pro Mini and a temperature sensor. It stores the temperature for a week when sampled every minute, without extra memory hardware.
The ATmega328P still getting such very useful feature is awesome.
// Test with Arduino Uno to test new optiboot feature to store data into flash.
// http://forum.arduino.cc/index.php?topic=332191.0
//
// An Arduino Pro Mini could be used to store the temperature for a week.
// The sketch requires manual start. An automatic start would be more useful.
#include "optiboot.h"
#include <avr/pgmspace.h>
// SPM_PAGESIZE for ATmega328P is 128
// Sampling every minute for a single channel with 160 pages is 10240 integer samples.
// That is one week storage.
// Now testing with 210 pages.
// Storage area in flash, only writable from the boot sector (using special optiboot bootloader).
// The compiler sets the whole contents default to zero.
const char flash_buffer[SPM_PAGESIZE * 210] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM = { 0,};
// Temporary storage in ram, to get a page of data.
uint8_t ram_buffer[SPM_PAGESIZE];
uint16_t *pSampleBuffer = (uint16_t *) ram_buffer; // create integer pointer to the ram buffer
boolean flagSampling = false;
int index; // sample index
unsigned long previousMillis;
// helper constants
const int MaxPageSamples = SPM_PAGESIZE / sizeof( uint16_t); // max integer samples in a single page.
const int MaxTotalSamples = sizeof( flash_buffer) / sizeof( uint16_t); // total max samples
const int Pages = sizeof(flash_buffer) / SPM_PAGESIZE; // number of storage pages.
const unsigned long interval = 60000UL; // 60000UL is one minute, try 200UL for testing
void setup() {
// Init serial
Serial.begin(9600); // increase to 115200 if dumping data takes very long.
Serial.println(F("\nSample analog A0 and store in flash"));
Serial.println(F("Storage:"));
Serial.print(F(" Samples to store : "));
Serial.println( MaxTotalSamples);
Serial.print(F(" Interval : "));
Serial.print( interval);
Serial.println(F(" ms"));
Serial.print(F(" Page size : "));
Serial.println(SPM_PAGESIZE);
Serial.print(F(" Available pages : "));
Serial.println( Pages);
Serial.println(F("Commands:"));
Serial.println(F(" s = start sampling"));
Serial.println(F(" c = clear storage area"));
Serial.println(F(" d = dump data"));
Serial.println(F(" q = stop sampling"));
}
void loop() {
int i;
if ( flagSampling) {
if ( millis() - previousMillis >= interval) { // software timer with millis()
previousMillis += interval;
// The 'index' counts the number of integer samples.
int data = analogRead( A0);
pSampleBuffer[index % MaxPageSamples] = data;
Serial.print(F("Sample "));
Serial.print(index);
Serial.print(F(" is "));
Serial.println( data);
if (((index + 1) % MaxPageSamples) == 0) { // was this the last sample for a page ?
Serial.print(F("Writing page "));
Serial.println( index / MaxPageSamples);
WriteRamToFlash( ram_buffer, index / MaxPageSamples);
}
index++;
if ( index >= MaxTotalSamples) {
// out of storage space
Serial.println(F("Storage space filled. Sampling stopped"));
flagSampling = false;
}
}
}
if ( Serial.available()) {
int inChar = Serial.read();
switch ( inChar) {
case 's':
Serial.println(F("Sampling started !"));
flagSampling = true;
index = 0;
previousMillis = millis(); // start software timer with millis with current millis()
break;
case 'c':
Serial.print(F("Clearing complete storage area... "));
memset( ram_buffer, '\0', sizeof(ram_buffer)); // fill ram buffer with zeros
for ( i = 0; i < Pages; i++)
WriteRamToFlash( ram_buffer, i);
Serial.println(F("Done"));
break;
case 'd':
// Print current flash buffer content
int n;
if ( index == 0) {
Serial.println(F("no new samples stored, dumping previous stored samples"));
n = MaxTotalSamples;
} else {
n = index;
}
for ( i = 0; i < n; i++) {
if ( i % 16 == 0)
Serial.println();
Serial.print( pgm_read_word( flash_buffer + (sizeof(int16_t) * i)));
Serial.print(F(", "));
}
Serial.println();
break;
case 'q':
Serial.println(F("Stopped sampling"));
flagSampling = false;
// Write current buffer, clear yet unwritten samples.
for( i=index % MaxPageSamples; i<MaxPageSamples; i++)
pSampleBuffer[i] = 0;
Serial.print(F("Writing page "));
Serial.println( index / MaxPageSamples);
WriteRamToFlash( ram_buffer, index / MaxPageSamples);
break;
}
}
}
// WriteRamToFlash
// This function writes a complete SPM_PAGESIZE buffer from ram to flash
// The page is the flash page (used as page offset to 'flash-buffer').
void WriteRamToFlash( uint8_t *buf, int page) {
optiboot_page_erase((uint16_t)(void*) &flash_buffer[page * SPM_PAGESIZE]);
for ( int i = 0; i < SPM_PAGESIZE; i += 2) { // 'i' is in bytes, write words
uint16_t w = word(buf[i+1], buf[i]); // word(high,low)
optiboot_page_fill((uint16_t)(void*) &flash_buffer[(page * SPM_PAGESIZE) + i], w);
}
optiboot_page_write((uint16_t)(void*) &flash_buffer[page * SPM_PAGESIZE]);
}
Hi majek,
This is great! If it's not too much trouble, could you please give a high level overview of this, such as what logic you have added to optiboot, and why the bootloader is involved with an application writing to flash?
I think the ATmega chip has a protection, so it is not possible to write to the flash from the sketch. Only from the high memory in the boot section it is allowed to write to flash.
The functions to write to flash are in the bootloader anyway, so majek made a fixed vector in the boot sector. That vector is do_spm, and everything is done via a call to that vector.
In the bootloader itself, that do_spm vector can execute the flash programming functions.
It is very straightforward, but far too complicated for me. So it’s an awesome piece of code. I send a tip to hackaday.com, that for the first time an Arduino can store data in its own flash from the sketch, runtime. It is as if every Arduino has suddenly extra free non-volatile memory to store things. Here is the hackaday blog post : http://hackaday.com/2015/07/03/arduinos-and-other-avrs-write-to-own-flash
So if I understand that correctly, the bootloader itself was not really modified at all, but some additional code which simply needs to co-reside with the bootloader is useful for reaching back into the application memory to do writes. And the bootloader plus co-resident additional code remains small enough to fit within the bounds of the small footprint of bootloader area?
Writing to flash is possible on AVR only by 'spm' instruction. This instruction doesn't work outside of bootloader section.
So, to write to flash, application needs to call function in bootloader section.
I could be done by simply adding such function to bootloader, but it adds many bytes to bootloader size and most bootloaders couldn't fit into it's section anymore after such modifications. Optiboot itself used 'spm' instruction simply 3 times just in 'inline' way. So, this was a chance for me I moved all 3 occurences of using 'spm' into separate function and replaced them by calls. This way I made function for application used also by bootloader. Such rewrite gave only few bytes larger code, that still fits into bootloader section.
Making header file and example was just proof of concept and it helped me to remove some bugs in first versions of this rewrite.
That is very crafty. I like mission impossible stuff. I wonder if the changes to the bootloader are appropriate to incorporate into he main optiboot and consider it a feature, and eventually make its way into mainstream Arduino perhaps as a library, or if this is too far out in the fringes to consider doing that.
I hope it will get in the official bootloader, since it is a useful feature.
majek, would it be possible to tell if a bootloader supports the new feature ? Before calling the vector Perhaps checking for specific bytes ?
Optiboot with this feature has two 'rjmp' instructions at the beginning, so at +1 is 0xC0 and at +3 it's 0xC? (on 512B bootloaders it's still 0xC0, on larger it could be also 0xC1, so it's safer to just check for 0xC? which is 'rjmp' opcode).
But if you have 'safe' lock bits set that 'lpm' function not allowed to read bootloader section, it's not possible to check this. And I see that this is default setting in Arduino, but luckily not in Optiboot.
You can jump also in blind Optiboot without this will just reboot after (or even without) timeout, most likely with only WDR flag set in MCUSR.
I don't know if I can do it, it seems complicated 
Perhaps checking for specific bytes ?