Go Down

Topic: Writing to FLASH from application. Please test and enjoy :-) (Read 21058 times) previous topic - next topic

majek

Hello.

I've made a modification to Optiboot bootloader, so now application can write to flash if this version of bootloader is installed.

Modified bootloader with precompiled hex files is available at https://github.com/majekw/optiboot/tree/supermaster
Installation is exactly the same as in original Optiboot.

Example is also provided (flash_program).

I hope you enjoy this long waited feature :-)

Of course, report of success or failure would be nice :-) I tested it on Atmega8, Atmega168 and Atmega328 without problems, but more testing could be good before it could be merged into next official release of Optiboot.

Marek

Peter_n

Could a sketch accept a gcc hex file and update the sketch via usb-serial ? Bypassing avrdude ? Via Ethernet or anything ?

majek

Yes, it's possible but very difficult if you want to reprogram it itself. But I know one project with its own bootloader capable to reprogram whole itself (including bootloader), so it's not impossible :-)

The main goal of writing flash by application is to store in non-volatile memory more data than in small eeprom. Sometimes you need only few kB of such storage and connecting external flash or sd-card is overkill.

For example I use it in my project on Atmega168/328, where application code is about 8KB and the rest of flash is used for data which application uses and can modify from time to time.

Of course, you must keep in mind that flash has it's limitations as less write/erase cycles or writing only in pages, but it's still good storage for more data not frequently written.

And it's dangerous for application if you make mistake in alligning write data, but it's also easier to find than out-of-the-blue jump to bootloader code destroing random part of flash :-)

Peter_n

#3
Jun 26, 2015, 10:07 am Last Edit: Jun 26, 2015, 10:07 am by Peter_n
For banks of data. Okay. But it is scary :o  I don't know which flash page is available.

I would like a full functioning diagnostics webpage on my Arduino Mega with Ethernet shield if the SD fails. That webpage changes when more things are added. I would like to update that page with the html file on my computer. So you granted my wish via the bootloader :P I don't know if I can do it, it seems complicated  :smiley-sad:

majek

It's not so scary :-) And for updating static html page from time to time, it's just perfect :-)

Check example in https://github.com/majekw/optiboot/blob/supermaster/optiboot/examples/flash_program/flash_program.ino for using flash_buffer variable.

You don't need to know any adresses - just allocate some memory for array containing your static html page using PROGMEM directive with additional __attribute__ (( aligned(SPM_PAGESIZE) )) and make it large enough to allow future updates. And it's size needs to be multiple of SPM_PAGESIZE.

Peter_n

It is a Mega 2560 board (SPM_PAGESIZE=256, EEPROM only 4kbyte).
The html file is larger than sram memory is available. So the flash programming could be done in chunks of 256 bytes. I think that is no problem, just increment the pointer with 256 for every chunk.
The flash_buffer may not be larger than 32kbyte. It seems to be a limit of the compiler. That's no problem, about 16kbyte is would be okay for me.
I don't know how to upload a file in chunks of 256 bytes, without serial handshake.

The flash programming is an awesome and unique feature, no doubt about that ! (but it is hard for me to implement).

majek

I don't know how to upload a file in chunks of 256 bytes, without serial handshake.
Split it into chunks before uploading (by hand or some javascript) and put into application only simple http endpoint which gets chunk number and 256B of data. I think it's much easier than dealing with it on chip side.

Peter_n

Hmmm, uploading a file with javascript in chunks of 256byte, and disabling interrupts while flash programming. It should be possible. I don't know if I have to the time to do a test.

majek

@Peter_n, i have bad news for you, Optiboot supports chips with at most 128KB flash, so it wont work on 2560 :-(
I didn't spotted it earlier...

I think that probably it could be adapted to work on 2560 without rewriting everything, but it would be still limited by old protocol to writing only to first 128KB of flash (bootloader itself, not my modification).

Peter_n

Tested on a Arduino Uno SMD, and working very well :P

A few notes:
  • It is not possible to keep the contents after uploading the sketch (because the compiler creates the buffer in the flash and defines the address)
  • For a page size of 128:
        page erase: 1.1ms
        page fill: 372 us (the whole loop)
        page write: 120us
    That is 12us per byte or 80kbyte per second.
  • It is allowed to turn off the interrupts with "noInterrupt()" and "interrupt()", thus no special precautions are needed for the rest of the sketch.
  • Recording analog data with 1kHz sample rate, and 20kbyte flash_buffer can record 20 seconds. When it is a loop recorder, it will reach the maximum allowed write/erase cycles after 189 days. That is assuming that the data is equally spread over 160 pages.
  • A good way to fill it, is for example a raw file on a SD card with audio or other sampled data.

majek

Thank you for testing!

A few notes to notes :-)

1. It's possible to keep contents after uploading new sketch, but not using predefined const variable like in the example.
It's possible to use any address between sketch code (aligned to page size) and bootloader. It requires some voodoo casting pointers, but in my opinion it's also safer for application (think about common 'off by one' errors and buffer placed between interrupt vectors and code, as in code generated by example).

3. Yes, functions in optiboot.h take care of disabling and enabling interrupts as needed in critical sections, so application could use it without special effort.

Peter_n

Could you try the next sketch. It is still quick and dirty, but it seems to work.
The sketch could be used with a small Pro Mini and a temperature sensor. It stores the temperature for a week when sampled every minute, without extra memory hardware.
The ATmega328P still getting such very useful feature is awesome.

Code: [Select]

// Test with Arduino Uno to test new optiboot feature to store data into flash.
//    http://forum.arduino.cc/index.php?topic=332191.0
//
// An Arduino Pro Mini could be used to store the temperature for a week.
// The sketch requires manual start. An automatic start would be more useful.


#include "optiboot.h"
#include <avr/pgmspace.h>

// SPM_PAGESIZE for ATmega328P is 128
// Sampling every minute for a single channel with 160 pages is 10240 integer samples.
// That is one week storage.
// Now testing with 210 pages.

// Storage area in flash, only writable from the boot sector (using special optiboot bootloader).
// The compiler sets the whole contents default to zero.
const char flash_buffer[SPM_PAGESIZE * 210] __attribute__ (( aligned(SPM_PAGESIZE) )) PROGMEM = { 0,};

// Temporary storage in ram, to get a page of data.
uint8_t ram_buffer[SPM_PAGESIZE];
uint16_t *pSampleBuffer = (uint16_t *) ram_buffer;    // create integer pointer to the ram buffer

boolean flagSampling = false;
int index;                         // sample index
unsigned long previousMillis;

// helper constants
const int MaxPageSamples = SPM_PAGESIZE / sizeof( uint16_t);  // max integer samples in a single page.
const int MaxTotalSamples = sizeof( flash_buffer) / sizeof( uint16_t);  // total max samples
const int Pages = sizeof(flash_buffer) / SPM_PAGESIZE;      // number of storage pages.
const unsigned long interval = 60000UL;                     // 60000UL is one minute, try 200UL for testing


void setup() {
  // Init serial
  Serial.begin(9600);             // increase to 115200 if dumping data takes very long.
  Serial.println(F("\nSample analog A0 and store in flash"));

  Serial.println(F("Storage:"));
  Serial.print(F("  Samples to store : "));
  Serial.println( MaxTotalSamples);
  Serial.print(F("  Interval : "));
  Serial.print( interval);
  Serial.println(F(" ms"));
  Serial.print(F("  Page size : "));
  Serial.println(SPM_PAGESIZE);
  Serial.print(F("  Available pages : "));
  Serial.println( Pages);

  Serial.println(F("Commands:"));
  Serial.println(F("  s = start sampling"));
  Serial.println(F("  c = clear storage area"));
  Serial.println(F("  d = dump data"));
  Serial.println(F("  q = stop sampling"));
}


void loop() {
  int i;

  if ( flagSampling) {
    if ( millis() - previousMillis >= interval) {           // software timer with millis()
      previousMillis += interval;

      // The 'index' counts the number of integer samples.
      int data = analogRead( A0);
      pSampleBuffer[index % MaxPageSamples] = data;

      Serial.print(F("Sample "));
      Serial.print(index);
      Serial.print(F(" is "));
      Serial.println( data);

      if (((index + 1) % MaxPageSamples) == 0) {           // was this the last sample for a page ?
        Serial.print(F("Writing page "));
        Serial.println( index / MaxPageSamples);
        WriteRamToFlash( ram_buffer, index / MaxPageSamples);
      }

      index++;

      if ( index >= MaxTotalSamples) {
        // out of storage space
        Serial.println(F("Storage space filled. Sampling stopped"));
        flagSampling = false;
      }
    }
  }
 
  if ( Serial.available()) {
    int inChar = Serial.read();
    switch ( inChar) {
      case 's':
        Serial.println(F("Sampling started !"));
        flagSampling = true;
        index = 0;
        previousMillis = millis();                        // start software timer with millis with current millis()
        break;

      case 'c':
        Serial.print(F("Clearing complete storage area... "));
        memset( ram_buffer, '\0', sizeof(ram_buffer));    // fill ram buffer with zeros
        for ( i = 0; i < Pages; i++)
          WriteRamToFlash( ram_buffer, i);
        Serial.println(F("Done"));
        break;

      case 'd':
        // Print current flash buffer content
        int n;
        if ( index == 0) {
          Serial.println(F("no new samples stored, dumping previous stored samples"));
          n = MaxTotalSamples;
        } else {
          n = index;
        }
       
        for ( i = 0; i < n; i++) {
          if ( i % 16 == 0)
            Serial.println();
          Serial.print( pgm_read_word( flash_buffer + (sizeof(int16_t) * i)));
          Serial.print(F(", "));
        }
        Serial.println();
        break;

      case 'q':
        Serial.println(F("Stopped sampling"));
        flagSampling = false;
        // Write current buffer, clear yet unwritten samples.
        for( i=index % MaxPageSamples; i<MaxPageSamples; i++)
          pSampleBuffer[i] = 0;

        Serial.print(F("Writing page "));
        Serial.println( index / MaxPageSamples);
        WriteRamToFlash( ram_buffer, index / MaxPageSamples);
        break;
    }
  }
}

// WriteRamToFlash
// This function writes a complete SPM_PAGESIZE buffer from ram to flash
// The page is the flash page (used as page offset to 'flash-buffer').
void WriteRamToFlash( uint8_t *buf, int page) {

  optiboot_page_erase((uint16_t)(void*) &flash_buffer[page * SPM_PAGESIZE]);

  for ( int i = 0; i < SPM_PAGESIZE; i += 2) {    // 'i' is in bytes, write words
    uint16_t w =  word(buf[i+1], buf[i]);            // word(high,low)
    optiboot_page_fill((uint16_t)(void*) &flash_buffer[(page * SPM_PAGESIZE) + i], w);
  }

  optiboot_page_write((uint16_t)(void*) &flash_buffer[page * SPM_PAGESIZE]);
}

dmjlambert

Hi majek,
This is great!  If it's not too much trouble, could you please give a high level overview of this, such as what logic you have added to optiboot, and why the bootloader is involved with an application writing to flash?

Peter_n

I think the ATmega chip has a protection, so it is not possible to write to the flash from the sketch. Only from the high memory in the boot section it is allowed to write to flash.

The functions to write to flash are in the bootloader anyway, so majek made a fixed vector in the boot sector. That vector is do_spm, and everything is done via a call to that vector.

In the project file (in the sketch) this optiboot.h must be included, and it translates the functions to that do_spm vector call : https://github.com/majekw/optiboot/blob/supermaster/optiboot/examples/flash_program/optiboot.h

In the bootloader itself, that do_spm vector can execute the flash programming functions.
https://github.com/majekw/optiboot/blob/supermaster/optiboot/bootloaders/optiboot/optiboot.c

It is very straightforward, but far too complicated for me. So it's an awesome piece of code. I send a tip to hackaday.com, that for the first time an Arduino can store data in its own flash from the sketch, runtime. It is as if every Arduino has suddenly extra free non-volatile memory to store things. Here is the hackaday blog post : http://hackaday.com/2015/07/03/arduinos-and-other-avrs-write-to-own-flash

dmjlambert

So if I understand that correctly, the bootloader itself was not really modified at all, but some additional code which simply needs to co-reside with the bootloader is useful for reaching back into the application memory to do writes.  And the bootloader plus co-resident additional code remains small enough to fit within the bounds of the small footprint of bootloader area?

Go Up