Help programming .hex bytes to flash

I am trying to write a bootloader to load a program from an EEPROM into flash. I have read Atmel notes AVR068, AVR109 and AVR910 but they mainly refer to the STK500 protocol which is not applicable to my problem.

I need to know which bytes from the .hex file are loaded into flash. An example of a very short hex file is as follows:

:10010000214601360121470136007EFE09D2190140 (Data record)
:00000001FF (End of file record)

Breaking it up into its constituent parts:

Data Record
Start code- :
Byte count- 10
Address- 0100
Record type- 00
Data- 214601360121470136007EFE09D21901
Checksum- 40

End of file record
Start code- :
Byte count- 00
Address- 0000
Record type- 01
Data: (Blank)
Checksum- FF

When copying these bytes into flash (starting at address 0x0000) I would start at the start of the hex file and copy the bytes in sequentially.

However, I assume that I would make the following exceptions:

  • not copy the the start code
  • not copy the byte count
  • do I copy the two address bytes in as stored in the hex file or do I need to switch them (big endian vs little endian) or do I need to copy them at all?
  • not copy record type
  • not copy checksum
  • not copy the CR/LF combination at the end of each hex record.

Can someone verify my assumptions and shed some light on the endian issue with the address?
Do I copy the end of file record to flash?
Do I insert a delimiter between writing each record?

lemming:
I am trying to write a bootloader to load a program from an EEPROM into flash. I have read Atmel notes AVR068, AVR109 and AVR910 but they mainly refer to the STK500 protocol which is not applicable to my problem.

I need to know which bytes from the .hex file are loaded into flash. An example of a very short hex file is as follows:

:10010000214601360121470136007EFE09D2190140 (Data record)
:00000001FF (End of file record)

Breaking it up into its constituent parts:

Data Record
Start code- :
Byte count- 10
Address- 0100
Record type- 00
Data- 214601360121470136007EFE09D21901
Checksum- 40

End of file record
Start code- :
Byte count- 00
Address- 0000
Record type- 01
Data: (Blank)
Checksum- FF

When copying these bytes into flash (starting at address 0x0000) I would start at the start of the hex file and copy the bytes in sequentially.

However, I assume that I would make the following exceptions:

  • not copy the the start code
  • not copy the byte count
  • do I copy the two address bytes in as stored in the hex file or do I need to switch them (big endian vs little endian) or do I need to copy them at all?

Don’t copy at all.

lemming:

  • not copy record type
  • not copy checksum
  • not copy the CR/LF combination at the end of each hex record.

That’s all correct.

lemming:
Can someone verify my assumptions and shed some light on the endian issue with the address?
Do I copy the end of file record to flash?

No.

lemming:
Do I insert a delimiter between writing each record?

No.

All your assumptions are otherwise correct.

Care to explain a bit further the motivation of storing a program in eeprom (short program!) before loading via bootloader into flash? Just curious.

So all I copy is the data section of each record in a sequention fashion to flash starting at 0x0000?

Why is there an address field in the record?

Care to explain a bit further the motivation of storing a program in eeprom

I have a description of what I am trying to achieve in the attached document below. I sent this document to a few luminaries on the Arduino and avrFreaks websites when seeking someone to write the bootloader; to no avail. I feel a bit out of my depth here.

BootloaderSpec3.txt (3.32 KB)

lemming:
So all I copy is the data section of each record in a sequention fashion to flash starting at 0x0000?

Well, starting at the first block after the reserved bootloader blocks (otherwise you will overwrite your bootloader).

lemming:
Why is there an address field in the record?

It provides information as to where the data is to be placed in memory. But it is metadata, and is not actually written to memory itself.

lemming:

Care to explain a bit further the motivation of storing a program in eeprom

I have a description of what I am trying to achieve in the attached document below. I sent this document to a few luminaries on the Arduino and avrFreaks websites when seeking someone to write the bootloader; to no avail. I feel a bit out of my depth here.

OK, I see you are using an external eeprom chip to store your sketch.

Note that writing to AVR program flash is done in blocks -- blocksize from 512 bytes, I believe, and maybe 1024 or 2048 depending on the chip. So your bootloader has to take that into account.

Well, starting at the first block after the reserved bootloader blocks (otherwise you will overwrite your bootloader).

I thought that the bootloader was high in memory (1F000 for atmega2560) and applications always started at 0x0000 (whether bootloader was installed or not.)

It provides information as to where the data is to be placed in memory

Does this mean that I should not just copy the data fields sequentially into memory but put them in the specific memory locations specificied by the address field?

Note that writing to AVR program flash is done in blocks

I understand that the Atmega2560 has a page size of 128 words, so I would load the Program Memory Page buffer before issuing the Write Program Memory Page instruction. I assume that it does not matter if a .hex file data field is written across a page boundary.

lemming:

Well, starting at the first block after the reserved bootloader blocks (otherwise you will overwrite your bootloader).

I thought that the bootloader was high in memory (1F000 for atmega2560) and applications always started at 0x0000 (whether bootloader was installed or not.)

Yes, that rings a bell now you mention it. (Looks at datasheet) When the boot loader fuse is programmed the processor starts at boot flash rather than starting at address 0. (So when the fuse is set, bootloader will always run at start/reset, and has to pass control to the main program when/if appropriate.)

So yes, you are right, start writing at address 0.

lemming:

It provides information as to where the data is to be placed in memory

Does this mean that I should not just copy the data fields sequentially into memory but put them in the specific memory locations specificied by the address field?

Yes. You will also have to check out how to read address > 64K for the 2560:

wikipedia:
04, Extended Linear Address Record, allowing for fully 32 bit addressing (up to 4GiB). The address field is 0000, the byte count is 02. The two data bytes (two hex digit pairs in big endian order) represent the upper 16 bits of the 32 bit address for all subsequent 00 type records until the next 04 type record comes. If there is not a 04 type record, the upper 16 bits default to 0000. To get the absolute address for subsequent 00 type records, the address specified by the data field of the most recent 04 record is added to the 00 record addresses.

lemming:

Note that writing to AVR program flash is done in blocks

I assume that it does not matter if a .hex file data field is written across a page boundary.

I wouldn't assume that; I believe you have to write the pages starting at the page boundaries.

So all I copy is the data section of each record in a sequention fashion to flash starting at 0x0000?

No, you should copy the data section of each record to flash at the ADDRESS that was IN the record. Those will normally be sequential, but they don't NEED to be. You might want to look at "optiloader" ( GitHub - WestfW/OptiLoader: Arduino sketch for burning new bootloaders ), which reads .hex format data from flash memory for purposes of programming a bootloader...

Thanks guys.

@westfw
Thanks for the link to your OptiLoader (a handy little tool too) but it is writing to the target through SPI, I assume using STK500 protocol. I am reading from an external EEPROM using SPI but writing 'locally' to the flash from the bootloader on the same device that is being programmed; which makes the example less valid.

I am trawling through the >2000 lines of the standard STK500v2 bootloader to see how they copy the bytes into flash in that program. The waters are muddied by all the hundreds of lines of serial comms and state machine which are woven through the flash programming logic. It would have been easy if the bootloader serially downloaded the hex in one section of code and then, in another function, just cycled through the the collected hex putting it into flash. As it is, the state machine\ fault tolerance and multitude of various micros catered for and the hundreds of lines long switch blocks, all in a single function, make the code difficult to understand. Its jumping around like spaghetti code, but then again, I suppose thats what state machines are about.

All I want is the code to write one of your progmem images into flash correctly and directly (not using SPI). I can then adapt it to read the hex from the ex ternal EEPROM.

Given my example hex file of:

:10010000214601360121470136007EFE09D2190140 (Data record)
:00000001FF (End of file record)

would the last byte of my program in flash be 0x01. i.e. no termination character indicating the end of program and no padding out to the page boundary?

The standard bootloaders don't deal with .hex files at all; the .hex file is converted to binary on the PC side (by avrdude) and the addresses are converted as required by the protocol (stk500v1 or stk500v2.) So looking at that code isn't going to be helpful for interpreting the hex files. Optiloader, OTOH, actually stores its images in the standard .hex format and includes code for reading it.

Programming the flash has to happen in chunks of the device page size, regardless of whether the .hex file has enough data to fill that page (the bootloaders generally have a bug here; they'll leave random contents in the unspecified bytes (if there isn't enough data to fill a page), instead of filling with 0xFF ("unprogrammed.") (actually, I keep claiming that it's a combination of bootloader and avrdude "cooperating" to create a bug :-()

The standard bootloaders don't deal with .hex files at all; the .hex file is converted to binary on the PC side

Does this mean that if I copy the hex values into flash at the addresses indicated in the hex records, the program/sketch will not run?

lemming:

The standard bootloaders don't deal with .hex files at all; the .hex file is converted to binary on the PC side

Does this mean that if I copy the hex values into flash at the addresses indicated in the hex records, the program/sketch will not run?

I think westfw is simply saying that your bootloader has to convert the hex to binary, since you will be reading a hex file file rather than a binary file from the eeprom. So your bootloader reads "FF" (two ascii characters, two bytes) it writes 0xff (one byte to flash).

Of course, one thing you could consider is sending the binary image to the eeprom, rather than the .hex file, in which case you would have to do the .hex to binary conversion before you transmit, rather than by the bootloader. Main advantage would be sending only ~1/2 the bytes compared to .hex, and your eeprom could be correspondingly smaller capacity to store the binary rather than the .hex representation.