under 1k bootloader, any interest?

I've been investigating options for getting the standard bootloader below 1k (currently it is 2k), but it is the sort of thing that is hard to get folks excited about.

Currently all arduinos and clones are using 2k of flash that could theoretically be done with one k of flash, freeing up 1k for all users who use it. This requires getting the chip sellers and arduino
IDE folks to come to the table and cooperate, as well as all the work testing and making extra sure the solution works.

I have a working model bootloader that is under 1k (and could be ported to the 328) and associated java based loader utility, so it isn't a theoretical thing, it is very doable, but currently requires the custom java loader as the small bootloader only understands a subset of the stk500 commands.

So I'm trying to decide if it is worth it to keep harping on this to not let 1k of flash on most every single arduino and clone ever made go unchallenged, or just let it die. You tell me (and anyone else who might be listening).

Here's the test subjects, loader runs standalone but pretty simple to integrate into the ide:

/* java based uploader for arduino and any other %100 open source projects,
 might also work with stk500 by chance  */

//uses rxtx http://users.frii.com/jarvi/rxtx/download.html

//courtesy dcb AT opengauge.org

import gnu.io.CommPortIdentifier;
import gnu.io.SerialPort;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.Method;

public class ArduinoLoader {

      public static final long rxtimeoutms = 5000;

      public static final byte[] hello = { 0x30, 0x20 };

      public static final byte[] okrsp = { 0x14, 0x10 };

      public static int startaddr = 0x00;
      
      static InputStream input2;

      static OutputStream output2;

      public static int waitForChar(InputStream i) throws Exception {
            long t = System.currentTimeMillis();
            while (i.available() == 0) {
                  try {
                        Thread.sleep(50);
                  } catch (Exception e) {
                  }
                  if (System.currentTimeMillis() - t > rxtimeoutms) {
                        throw new Exception("Timed out waiting for response");
                  }
            }
            return i.read();
      }

      public static void chkok(InputStream i) throws Exception {
            if ((waitForChar(i) != okrsp[0]) || (waitForChar(i) != okrsp[1]))
                  throw new Exception("Invalid response");
      }


      static int[] image = new int[200000];

      static int imagesize = 0;

      // pass hex file, com port ID, baud, page size
      public static void main(String[] args) throws Exception {
            long g = System.currentTimeMillis(); 
//            args = new String[] { "/Blink.hex", "COM4", "19200", "128" };
            if (args.length != 4) {
                  System.out
                              .println("Arguments: full_path_to_hex_file com_port_id baud page_size");
                  System.exit(1);
            }


            
            upload(args[0], args[1], Integer.parseInt(args[2]), Integer
                        .parseInt(args[3]), System.out);
            g=System.currentTimeMillis()-g;
            System.out.println("Completed, " + imagesize + " bytes uploaded " + g);

      }

      static void println(String s, OutputStream o) throws Exception {
            o.write((s + "\n").getBytes());
      }

      public static void upload(String filename, String comport, int baud,
                  int pageSize, OutputStream out) throws Exception {
            println("Serial Proxy Starting, port:" + comport + ", baud:" + baud
                        + ", pagesize:" + pageSize + ", file:" + filename, out);

            // load the hex file into memory
            parsehex(filename);

            // out=new FileOutputStream("/arduinoloader.log");
            out = System.out;

            CommPortIdentifier portId2 = CommPortIdentifier
                        .getPortIdentifier(comport);

            SerialPort port2 = (SerialPort) portId2.open("serial madness2", 4001);
            input2 = port2.getInputStream();
            output2 = port2.getOutputStream();
            port2.setSerialPortParams(baud, SerialPort.DATABITS_8,
                        SerialPort.STOPBITS_1, SerialPort.PARITY_NONE);

            port2.setDTR(false);
            output2.write(hello);

            try {
                  Thread.sleep(100);
            } catch (InterruptedException e) {
            }

            while (input2.available() > 0) {
                  input2.read();
            }
            ;

            for (int x = 0; x < 3; x++) {
                  try {
                        output2.write(hello);
                        chkok(input2);
                  } catch (Exception e) {
                  }
                  try {
                        Thread.sleep(500);
                  } catch (Exception e) {
                  }
            }
            output2.write(hello);
            chkok(input2);

            // write the hex file
            int addr = 0;
            while (addr < imagesize) {
                  output2.write('U');
                  output2.write((addr / 2) % 256);
                  output2.write((addr / 2) / 256);
                  output2.write(' ');
                  chkok(input2);

                  int ps = Math.min(pageSize, imagesize - addr);

                  output2.write('d');
                  output2.write(ps / 256);
                  output2.write(ps % 256);
                  output2.write('F');
                  for (int x = 0; x < ps; x++) {
                        output2.write(image[addr + x]);
                  }
                  output2.write(' ');
                  chkok(input2);
                  addr += ps;
            }

            // validate the image
            addr = 0;
            output2.write('U');
            output2.write((addr / 2) % 256);
            output2.write((addr / 2) / 256);
            output2.write(' ');
            chkok(input2);

            output2.write('t');
            output2.write(imagesize / 256);
            output2.write(imagesize % 256);
            output2.write('F');
            output2.write(' ');

            if ((waitForChar(input2) != okrsp[0]))
                  throw new Exception("Invalid response");

            while (addr < imagesize) {
                  int c = waitForChar(input2);
                  if (c != image[addr])
                        throw new Exception("Validation error at offset " + addr
                                    + ".  Expected " + image[addr] + " received " + c);
                  addr++;
                  // System.out.print(hexval(c));
                  if (addr % 16 == 0)
                        System.out.println("");
            }

            if ((waitForChar(input2) != okrsp[1]))
                  throw new Exception("Invalid response");

            output2.write('Q');
            output2.write(' ');
            chkok(input2);

            port2.setDTR(true);
            port2.close();
      }

      public static void waitfor(int w) throws Exception {
            int c;
            do {
                  while (input2.available() == 0)
                        ;
                  c = input2.read();
                  System.out.println((char) c + " " + (int) c);
            } while (c != w);
      }

      static String hexvals[] = { "0", "1", "2", "3", "4", "5", "6", "7", "8",
                  "9", "A", "B", "C", "D", "E", "F" };

      static String hexval(int i) {
            return hexvals[i / 16] + hexvals[i % 16];
      }

      static void parsehex(String fname) throws Exception {
            BufferedReader in = new BufferedReader(new FileReader(fname));
            String t = in.readLine();
            int line = 0;
            imagesize = 0;
            while (t != null) {
                  line++;
                  if (!":".equals(t.substring(0, 1)))
                        throw new Exception("line#" + line
                                    + " Invalid format in hex file " + fname);
                  int len = Integer.parseInt(t.substring(1, 3), 16);
                  imagesize += len;
                  int addr = Integer.parseInt(t.substring(3, 7), 16);
                  int type = Integer.parseInt(t.substring(7, 9), 16);
                  String datav = t.substring(9, 9 + (len * 2));
                  int[] data = new int[datav.length() / 2];
                  for (int x = 0; x < data.length; x++) {
                        data[x] = Integer.parseInt(datav.substring(x * 2, x * 2 + 2),
                                    16);
                  }
                  int cksum = Integer.parseInt(t.substring(9 + (len * 2),
                              11 + (len * 2)), 16);

                  // compute checksum of line just read
                  int cks = (256 - len) + (256 - (addr / 256)) + (256 - (addr % 256))
                              + (256 - type);
                  for (int x = 0; x < data.length; x++) {
                        cks += (256 - data[x]);
                  }
                  cks %= 256;
                  if (cks != cksum)
                        throw new Exception("line#" + line
                                    + " Invalid checksum in hex file " + fname);

                  // copy to the image so we can work with the page size easier
                  for (int x = 0; x < data.length; x++) {
                        image[addr + x] = data[x];
                  }

                  t = in.readLine();
            }

      }

}

Here's the first half of the bootloader, compiled with -nostartfiles added and -Wl,--section-start=.text=0x3c00 with arduino 0011 tools, and with the maximum script size upped to 15360 and -Os

/**********************************************************/
/* Serial Bootloader for Atmel megaAVR Controllers        */
/*                                                        */
/* tested with ATmega8, ATmega128 and ATmega168           */
/* should work with other mega's, see code for details    */
/*                                                        */
/* ATmegaBOOT.c                                           */
/*                                                        */
/* 20090201: hacked for Java uploader                     */
/*           stripped out all unused commands/code        */
/*           swapped bulk of assembler for examples at:   */
/* http://www.stderr.org/doc/avr-libc/avr-libc-user-manual/group__avr__boot.html   */
/*           by D. Brink, dcb AT opengauge.org          */
/* 20070626: hacked for Arduino Diecimila (which auto-    */
/*           resets when a USB connection is made to it)  */
/*           by D. Mellis                                 */
/* 20060802: hacked for Arduino by D. Cuartielles         */
/*           based on a previous hack by D. Mellis        */
/*           and D. Cuartielles                           */
/*                                                        */
/*                                                        */
/* based on stk500boot.c                                  */
/* Copyright (c) 2003, Jason P. Kyle                      */
/* All rights reserved.                                   */
/* see avr1.org for original file and information         */
/*                                                        */
/* This program is free software; you can redistribute it */
/* and/or modify it under the terms of the GNU General    */
/* Public License as published by the Free Software       */
/* Foundation; either version 2 of the License, or        */
/* (at your option) any later version.                    */
/*                                                        */
/* This program is distributed in the hope that it will   */
/* be useful, but WITHOUT ANY WARRANTY; without even the  */
/* implied warranty of MERCHANTABILITY or FITNESS FOR A   */
/* PARTICULAR PURPOSE.  See the GNU General Public        */
/* License for more details.                              */
/*                                                        */
/* You should have received a copy of the GNU General     */
/* Public License along with this program; if not, write  */
/* to the Free Software Foundation, Inc.,                 */
/* 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA */
/*                                                        */
/* Licence can be viewed at                               */
/* http://www.fsf.org/licenses/gpl.txt                    */
/*                                                        */
/* Target = Atmel AVR m128,m64,m32,m16,m8,m162,m163,m169, */
/* m8515,m8535. ATmega161 has a very small boot block so  */
/* isn't supported.                                       */
/*                                                        */
/* Tested with m168                                       */
/**********************************************************/

/* $Id$ */


/* some includes */
#include <inttypes.h>
#include <avr/io.h>
#include <avr/pgmspace.h>
#include <avr/boot.h>
#include <avr/interrupt.h>
#include <avr/wdt.h>
#include <avr/eeprom.h>


/* the current avr-libc eeprom functions do not support the ATmega168 */
/* own eeprom write/read functions are used instead */
//#ifndef __AVR_ATmega168__
//#include <avr/eeprom.h>
//#endif

/* Use the F_CPU defined in Makefile */

/* 20070707: hacked by David A. Mellis - after this many errors give up and launch application */
#define MAX_ERROR_COUNT 5

/* set the UART baud rate */
#define BAUD_RATE   19200


/* other ATmegas have only one UART, so only one pin is defined to enter bootloader */
#define BL_DDR  DDRD
#define BL_PORT PORTD
#define BL_PIN  PIND
#define BL      PIND6


/* onboard LED is used to indicate, that the bootloader was entered (3x flashing) */
/* if monitor functions are included, LED goes on after monitor was entered */
/* Onboard LED is connected to pin PB2 (e.g. Crumb8, Crumb168) */
#define LED_DDR  DDRB
#define LED_PORT PORTB
#define LED_PIN  PINB
/* 20060803: hacked by DojoCorp, LED pin is B5 in Arduino */
/* #define LED      PINB2 */
#define LED      PINB5


#define PAGE_SIZE    0x40U    //64 words


/* function prototypes */
void putch(char);
char getch(void);
void nothing_response(void);
//void flash_led(uint8_t);

unsigned int pagenumber=0;

/* some variables */
union address_union {
    uint16_t word;
    uint8_t  byte[2];
} address;

union length_union {
    uint16_t word;
    uint8_t  byte[2];
} length;

struct flags_struct {
    unsigned eeprom : 1;
    unsigned rampz  : 1;
} flags;

uint8_t buff[256];
uint8_t address_high;

uint8_t pagesz=0x80;

uint8_t i;
uint8_t bootuart = 0;

uint8_t error_count = 0;

    void boot_program_page (uint32_t page, uint8_t *buf)
    {
        uint16_t i;
        uint8_t sreg;

        // Disable interrupts.

        sreg = SREG;
        cli();
   
        eeprom_busy_wait ();

        boot_page_erase (page);
        boot_spm_busy_wait ();      // Wait until the memory is erased.

        for (i=0; i<SPM_PAGESIZE; i+=2)
        {
            // Set up little-endian word.

            uint16_t w = *buf++;
            w += (*buf++) << 8;
       
            boot_page_fill (page + i, w);
        }

        boot_page_write (page);     // Store buffer in flash page.
        boot_spm_busy_wait();       // Wait until the memory is written.

        // Reenable RWW-section again. We need this if we want to jump back
        // to the application after bootloading.

        boot_rww_enable ();

        // Re-enable interrupts (if they were ever enabled).

        SREG = sreg;
    }


void (*app_start)(void) = 0x0000;
int main (void) __attribute__ ((naked,section (".init9")));
int main(void){
    asm volatile ( "clr __zero_reg__" );
    SP=RAMEND;

    uint8_t ch;
    uint16_t w;

    asm volatile("nop\n\t");

    /* set pin direction for bootloader pin and enable pullup */
    /* for ATmega128, two pins need to be initialized */
    /* We run the bootloader regardless of the state of this pin.  Thus, don't
    put it in a different state than the other pins.  --DAM, 070709
    BL_DDR &= ~_BV(BL);
    BL_PORT |= _BV(BL);
    */

    /* check if flash is programmed already, if not start bootloader anyway */
    if(pgm_read_byte_near(0x0000) != 0xFF) {

    /* check if bootloader pin is set low */
    /* we don't start this part neither for the m8, nor m168 */
    //if(bit_is_set(BL_PIN, BL)) {
    //      app_start();
    //    }
    }



    /* initialize UART(s) depending on CPU defined */
    UBRR0L = (uint8_t)(F_CPU/(BAUD_RATE*16L)-1);
    UBRR0H = (F_CPU/(BAUD_RATE*16L)-1) >> 8;
    UCSR0B = (1<<RXEN0) | (1<<TXEN0);
    UCSR0C = (1<<UCSZ00) | (1<<UCSZ01);

    /* Enable internal pull-up resistor on pin D0 (RX), in order
    to supress line noise that prevents the bootloader from
    timing out (DAM: 20070509) */
    DDRD &= ~_BV(PIND0);
    PORTD |= _BV(PIND0);

    /* set LED pin as output */
    LED_DDR |= _BV(LED);

2nd half of modified bootloader:

    /* flash onboard LED to signal entering of bootloader */
//    flash_led(NUM_LED_FLASHES);
   
    /* 20050803: by DojoCorp, this is one of the parts provoking the
                 system to stop listening, cancelled from the original */
    //putch('\0');


    /* forever loop */
    for (;;) {

    /* get character from UART */
    ch = getch();

    /* A bunch of if...else if... gives smaller code than switch...case ! */

    /* Hello is anyone home ? */
    if(ch=='0') {
        nothing_response();
    }



    /* Leave programming mode  */
    else if(ch=='Q') {
        nothing_response();
    }


    /* Set address, little endian. EEPROM in bytes, FLASH in words  */
    /* Perhaps extra address bytes may be added in future to support > 128kB FLASH.  */
    /* This might explain why little endian was used here, big endian used everywhere else.  */
    else if(ch=='U') {
        address.byte[0] = getch();
        address.byte[1] = getch();
        nothing_response();
    }


    /* Write memory, length is big endian and is in bytes  */
    else if(ch=='d') {
        length.byte[1] = getch();
        length.byte[0] = getch();
        flags.eeprom = 0;
        if (getch() == 'E') flags.eeprom = 1;
        for (w=0;w<length.word;w++) {
        buff[w] = getch();                            // Store data in buffer, can't keep up with serial data stream whilst programming pages
        }
        if (getch() == ' ') {
        if (flags.eeprom) {                        //Write to EEPROM one byte at a time
            for(w=0;w<length.word;w++) {
            eeprom_write_byte((void *)address.word,buff[w]);
            address.word++;
            }           
        }
        else {                            //Write to FLASH one page at a time
            boot_program_page(pagenumber,&buff);
            pagenumber+=SPM_PAGESIZE;
        }
        putch(0x14);
        putch(0x10);
        } else {
        if (++error_count == MAX_ERROR_COUNT)
            app_start();
        }       
    }
   

        /* Read memory block mode, length is big endian.  */
        else if(ch=='t') {
        length.byte[1] = getch();
        length.byte[0] = getch();
        if (getch() == 'E') flags.eeprom = 1;
        else {
        flags.eeprom = 0;
        address.word = address.word << 1;            // address * 2 -> byte location
        }
        if (getch() == ' ') {                        // Command terminator
        putch(0x14);
        for (w=0;w < length.word;w++) {                // Can handle odd and even lengths okay
            if (flags.eeprom) {                            // Byte access EEPROM read
            while(EECR & (1<<EEPE));
            EEAR = (uint16_t)(void *)address.word;
            EECR |= (1<<EERE);
            putch(EEDR);
            address.word++;
            }
            else {

            if (!flags.rampz) putch(pgm_read_byte_near(address.word));
            address.word++;
            }
        }
        putch(0x10);
        }
    }




    else if (++error_count == MAX_ERROR_COUNT) {
        app_start();
    }
    }
    /* end of forever loop */

}



void putch(char ch)
{
    while (!(UCSR0A & _BV(UDRE0)));
    UDR0 = ch;
}


char getch(void)
{
    uint32_t count = 0;
    while(!(UCSR0A & _BV(RXC0))){
        /* 20060803 DojoCorp:: Addon coming from the previous Bootloader*/               
        /* HACKME:: here is a good place to count times*/
        count++;
        if (count > MAX_TIME_COUNT)
            app_start();
     }
    return UDR0;
}


void nothing_response(void)
{
    if (getch() == ' ') {
    putch(0x14);
    putch(0x10);
    } else {
    if (++error_count == MAX_ERROR_COUNT)
        app_start();
    }
}


/* end of file ATmegaBOOT.c */

I should also cast about to see if anyone sees a way to make the bootloader compatable with avrdude as delivered and still under 1k.

If this modded bootloader can work without all the assembler stuff used in the stock bootloader, why not. So I'll understand it too :slight_smile: Maybe even throw out the assembler code in the stock bootloader if this one works well.

The lack of compatibility with avrdude on the other hand would be an obstacle for me. I like using 'standard' tools, and more importantly tools that run in a console. This may not be important for windoze users, but on linux/mac I simply don't see why I should have to run X just for flashing a chip. And with the Atmega328, lack of FLASH is even less pressing. So do I see this tiny bootloader replace the standard one: no. But it could be distributed with the IDE as an option for experts, and code improvements (fixes, removal of asm) could be fed into the old bootloader.

Just FYI, the java version can be run from the command line, just like in the java connectivity example in the playground, Arduino Playground - Java

Only you will need to add %1 %2 %3 %4 %5 %6 to the script and call it with something like:
testrun java ArduinoLoader ~/Blink.hex COM4 19200 128

Sorry for not replying earlier or on the developers list. I also think the lack of compatibility with avrdude is a problem for a few reasons. One, some people like to use Arduino boards without the software. Second, it means that the Java uploader has to be maintained. Also, which commands have been removed and what does avrdude use them for?

So what happened between the (1k) mega8 bootloader and the (2k) mega168 bootloader to account for the increase in size? I looked at the sources, but I'm having a hard time filtering out all the bits that don't get compiled for the 168 anyway (there's a monitor and debugger in there...)

David, the java version works with the existing bootloader (and probably with any serial/ftdi stk500 for arduino purposes). But I mainly created it because my chances of success were better with a scratch java program than figuring out how to compile avrdude or get a version that sent out less requests to the programmer.

For the under 1k version, I left just the following commands in the bootloader:
0, hello?
Q, quit
U, set address
d, write memory (eprom or flash)
t, read memory (eprom or flash)

If I add the following commands back into the bootloader, it works with avrdude as delivered, but is 0x442 bytes, and they are mostly dummy functions:
A, get hw,sw ver functions
V, spi command
B, device parms
E, parallel prog
P, program mode (upgrade)

These are the remaining commands that were implemented in the original bootloader, that presumably make it work with avrstudio or somesuch.
1, echo "AVR ISP"
@, board commands
R, erase device
u, device signature bytes
v, read oscillator bytes
!, monitor stuff (don't think this is being built)

Phil, I don't know the answer to your question. I suspect compiler changes is part of it, and perhaps the different chip and more functionality (auto-reset) are part of it.

If I recall correctly, avrdude can be configured for multiple protocols. I suspect that what you need to do is: 1) add your protocol to the avrdude program and 2) edit the config file to add your protocol. I think arduino has its own copy of avrdude, so getting your changes accepted by the arduino folks may not be as hard as getting them accepted by the avrdude folks.

NOTE: To be very clear, "If I recall correctly ...". This is based on a 2-3 month old recollection of a quick peek at the avrdude code ... I'm likely lying through my teeth ;->

any news here?

is there any checksum in the current bootloader or in the smaller version? I only see that a lot of bytes (a page or even more??) can be transmitted to the Arduino but there does not seem to be a checksum or echo of the bytes... is this correct?