EEMEM directive and EEPROM storage in .HEX files

If I declare a eeprom variable like this,

EEMEM unsigned char my_storage_byte;

The compiler wants to join EEPROM and FLASH sections into the same .HEX file. This normally would not be a problem, except when AVRdude gets to the end of the .HEX file and doesn't understand that the address he's reading is really an EEPROM address. AVRdude then gives up and crashes.

My guess is that EEPROM and FLASH sections are joined by the Arduino compiler into the same file. Can I tell it not to do this?

How have others dealt with this issue?

If you set build.verbose to true in your Arduino preferences file, and then run Arduino and compile your sketch, you'll see the command lines passed to avr-g++, etc. To be honest, I'm not sure how we're handling EEPROM sections, but if you've got specific suggestions for changes to the compilation command lines, let me know.

Is it possible to build separate .hex files and .eep files? That way AVRdude would understand.

The makefile would look have something like this in it:

## Intel Hex file production flags
HEX_FLASH_FLAGS = -R .eeprom

HEX_EEPROM_FLAGS = -j .eeprom
HEX_EEPROM_FLAGS += --set-section-flags=.eeprom="alloc,load"
HEX_EEPROM_FLAGS += --change-section-lma .eeprom=0 --no-change-warnings


## Build
all: $(TARGET) myHexFileName.hex myHexFileName.eep size

##Link
$(TARGET): $(OBJECTS)
       $(CC) $(LDFLAGS) $(OBJECTS) $(LINKONLYOBJECTS) $(LIBDIRS) $(LIBS) -o $(TARGET)

%.hex: $(TARGET)
      avr-objcopy -O ihex $(HEX_FLASH_FLAGS)  $< $@

%.eep: $(TARGET)
      -avr-objcopy $(HEX_EEPROM_FLAGS) -O ihex $< $@ || exit 0

So before you call to AVRdude, you would just check to see if there was a .eep file. If so, then automatically burn it to EEPROM?

Any reason why EEPROM data is being put into the same flash .HEX file if avrdude doesn’t understand it?

Not explicitly. It's never been an issue, so I've just been doing what's worked. Here's what I think are the relevant command lines:

hardware/tools/avr/bin/avr-gcc -Os -mmcu=atmega168 -o /tmp/build6214.tmp/ASCIITable.elf /tmp/build6214.tmp/Temporary_9591_5305.cpp.o /tmp/build6214.tmp/core.a -L/tmp/build6214.tmp -lm hardware/tools/avr/bin/avr-objcopy -O srec -R .eeprom /tmp/build6214.tmp/ASCIITable.elf /tmp/build6214.tmp/ASCIITable.rom hardware/tools/avr/bin/avr-objcopy -O ihex -R .flash /tmp/build6214.tmp/ASCIITable.elf /tmp/build6214.tmp/ASCIITable.hex

What would they need to be changed to in order to separate the EEPROM and Flash data?

I would suggest trying to change this,

hardware/tools/avr/bin/avr-objcopy -O ihex -R .flash /tmp/build6214.tmp/ASCIITable.elf /tmp/build6214.tmp/ASCIITable.hex

To this,

hardware/tools/avr/bin/avr-objcopy -O ihex -R .eeprom  /tmp/build6214.tmp/ASCIITable.elf /tmp/build6214.tmp/ASCIITable.hex 
hardware/tools/avr/bin/avr-objcopy -j .eeprom --set-section-flags=.eeprom="alloc,load" --change-section-lma .eeprom=0 --no-change-warnings -O ihex /tmp/build6214.tmp/ASCIITable.elf /tmp/build6214.tmp/ASCIITable.eep || exit 0

Let me know if that works. If not, we’ll look at adjusting the avr-gcc call as well.

On a separate note, wouldn’t it be nice to see the AVR memory usage (program/data/eeprom) breakdown right in the compiler output window? This feedback could be useful to some people.

To get the memory map,

you use this call,

avr-gcc.exe -mmcu=atmega168 -Wl,-Map=/tmp/build6214.tmp/ASCIITable.map /tmp/build6214.tmp/Temporary_9591_5305.cpp.o -o /tmp/build6214.tmp/ASCIITable.elf

Hmm, definitely something to think about, but probably not a high priority.

Also, I'm not sure if avrdude and our bootloader support writing to the EEPROM. Have you tried it?

Looking through the bootloader code briefly, it looks like AVRdude should be able to burn EEPROM.

But this is not really what I was looking for. I'm just looking to use this:

EEMMEM unsigned char my_var;

And not have it put blank EEPROM variables at the end of the .HEX file (and crash AVRdude)

The reason why you would want this functionality is so you don't have to keep track of where eeprom locations are. You can just create a variable, and the compiler figures out what EEPROM address it should go in. This gets very helpful when dealing with complex structures stored in EEPROM.

I'ld love to see support for the EEMEM directive too! Added to the SuggestionsBugs page on the playground.

I had a look at the makefile in /hardware/cores/arduino/Makefile, to see if I could get this working. Disclaimer: I don’t really know what I am doing.

With two very minor changes to the makefile, it creates an .eep file for me. The .hex file also has the correct filesize, and both the flash and eeprom sections seem to be uploaded. I have not tested the resulting uploads yet (but my arduino is still accepting new uploads, so at least I did not fry the bootloader ;-). Below is a patch:

--- /hardware/cores/arduino/Makefile.orig      Fri May 02 18:14:46 2008
+++ /hardware/cores/arduino/Makefile      Sun May 04 15:17:04 2008
@@ -95,6 +95,7 @@
 # Programming support using avrdude. Settings and variables.
 AVRDUDE_PORT = $(PORT)
 AVRDUDE_WRITE_FLASH = -U flash:w:applet/$(TARGET).hex
+AVRDUDE_WRITE_EEPROM = -U eeprom:w:applet/$(TARGET).eep
 AVRDUDE_FLAGS = -V -F -C $(INSTALL_DIR)/hardware/tools/avr/etc/avrdude.conf \
 -p $(MCU) -P $(AVRDUDE_PORT) -c $(AVRDUDE_PROGRAMMER) \
 -b $(UPLOAD_RATE)
@@ -127,7 +128,7 @@
 # Default target.
 all: applet_files build sizeafter
 
-build: elf hex 
+build: elf eep hex 
 
 applet_files: $(TARGET).pde
       # Here is the "preprocessing".
@@ -149,8 +150,8 @@
 sym: applet/$(TARGET).sym
 
 # Program the device.  
-upload: applet/$(TARGET).hex
-      $(AVRDUDE) $(AVRDUDE_FLAGS) $(AVRDUDE_WRITE_FLASH)
+upload: applet/$(TARGET).hex applet/$(TARGET).eep
+      $(AVRDUDE) $(AVRDUDE_FLAGS) $(AVRDUDE_WRITE_FLASH) $(AVRDUDE_WRITE_EEPROM)
 
 
       # Display size of file.
@@ -185,7 +186,7 @@
       $(OBJCOPY) -O $(FORMAT) -R .eeprom $< $@
 
 .elf.eep:
-      -$(OBJCOPY) -j .eeprom --set-section-flags=.eeprom="alloc,load" \
+      -$(OBJCOPY) -j .eeprom --set-section-flags=.eeprom=alloc,load \
       --change-section-lma .eeprom=0 -O $(FORMAT) $< $@
 
 # Create extended listing file from ELF output file.

edit: edited the patch to also handle uploading the eep section.

Awesome.

If someone wants to patch the IDE for this as well, that would be great. Otherwise, I'm afraid it's not too high on the priority list.

I can have a look if you point me at where the buildproces is in the IDE sources... Like with the makefile, I won't really know what I'll be doing, but I can poke at the code until it works ;-)

The patch above needs some work; it does not like it when there is no .eep file. Is there such a thing as an "elseif" statement in a makefile?

update: /app/Compiler.java and /app/AvrdudeUploader.java look like the most likely suspects. Time to brush up my java...

Looks like I'll have a patch next week. The changes are very minor, but I need to do some testing with real projects before I can vouch for the code...

Hmm, it seems that my initial enthusiasm was somewhat premature… It seems to work fine, until I declare an array in EEMEM, at which point the whole eeprom writing stage fails. I could use an extra pair of eyes…

Index: AvrdudeUploader.java

===================================================================

--- AvrdudeUploader.java      (revision 453)

+++ AvrdudeUploader.java      (working copy)

@@ -44,6 +44,7 @@

     } else {
       Collection params = getProgrammerCommands(Preferences.get("upload.using"));
       params.add("-Uflash:w:" + buildPath + File.separator + className + ".hex:i");
+      params.add("-Ueeprom:w:" + buildPath + File.separator + className + ".eep:i");
       return avrdude(params);
     }
   }
@@ -62,6 +63,7 @@

       "-b" + Preferences.getInteger("boards." + Preferences.get("board") + ".upload.speed"));
     commandDownloader.add("-D"); // don't erase
     commandDownloader.add("-Uflash:w:" + buildPath + File.separator + className + ".hex:i");
+    commandDownloader.add("-Ueeprom:w:" + buildPath + File.separator + className + ".eep:i");
 
     if (Preferences.get("boards." + Preferences.get("board") + ".upload.disable_flushing") == null ||
         Preferences.getBoolean("boards." + Preferences.get("board") + ".upload.disable_flushing") == false) {
Index: Compiler.java

===================================================================

--- Compiler.java      (revision 453)

+++ Compiler.java      (working copy)

@@ -211,16 +211,20 @@

       List commandObjcopy;
       
       commandObjcopy = new ArrayList(baseCommandObjcopy);
-      commandObjcopy.add(2, "srec");
+      commandObjcopy.add(2, "ihex");
+      commandObjcopy.set(3, "-j");
       commandObjcopy.add(".eeprom");
+      commandObjcopy.add("--set-section-flags=.eeprom=alloc,load");
+      commandObjcopy.add("--change-section-lma");
+      commandObjcopy.add(".eeprom=0");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
-      commandObjcopy.add(buildPath + File.separator + sketch.name + ".rom");
+      commandObjcopy.add(buildPath + File.separator + sketch.name + ".eep");
       if (execAsynchronously(commandObjcopy) != 0)
         return false;
 
       commandObjcopy = new ArrayList(baseCommandObjcopy);
       commandObjcopy.add(2, "ihex");
-      commandObjcopy.add(".flash");
+      commandObjcopy.add(".eeprom");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".hex");
       if (execAsynchronously(commandObjcopy) != 0)

The patch above changes the following:

  • Compiler.java is changed so it creates an .eep file in ihex format, instead of a .rom file in srec format
  • Compiler.java is changed so the .hex file does not contain the eeprom section
  • AvrdudeUploader.java is changed so it also uploads the .eep file to eeprom space

You can use this sketch to test the modifications:

#include <avr/eeprom.h>

byte EEMEM NonVolatileChar = 123;
int  EEMEM NonVolatileInt = 12345;
//char EEMEM NonVolatileString[15]; = "Hello world"; 

void setup() { 
    char SRAMchar;
    int SRAMint;
    char SRAMstring[15];

    SRAMchar = eeprom_read_byte(&NonVolatileChar);
    SRAMint  = eeprom_read_word(&NonVolatileInt);
    //eeprom_read_block((void*)&SRAMstring, (const void*)&NonVolatileString, 15); 

    Serial.begin(9600);
    Serial.println(SRAMchar,DEC);
    Serial.println(SRAMint,DEC);
    Serial.println(SRAMstring);
}

void loop() {
  
}

When used like this, the code works fine; The NonVolatileChar and NonVolatileInt are read as expected. Uncommenting the EEMEM declaration of NonVolatileString breaks the eeprom upload process for me, and not even the NonVolatileChar and NonVolatileInt read back correctly anymore. This is the same with both the IDE changes in this post and the makefile changes I posted before, though the makefile by default does not verify the written data.

Update: To my untrained eye, it looks like the bootloader is putting bytes in the wrong places in the eeprom section. Something about byte vs word boundaries and/or big vs little endian-ness. The bootloader is special-casing the Atmega 168 when it comes to writing the eeprom section, and I don’t understand that code. I’ll have to build me an AVR ISP so I can see what exactly gets put in the eeprom, and to see if it works if I use the AVR ISP instead of the bootloader.

Finally had the time to construct my usbtinyisp. Unfortunately, the bootloader seems to foobar the eeprom data during upload. The eeprom data seems to come across intact when I choose to use the usbtinyisp for programming the sketch, but the bootloader munges up a couple of bytes per handful of bytes.

So even if I could get this fixed, chances of it getting in are very slim, right? Seeing that it would need a bootloader update...

Either way, it would be great if the following patch could get in; it fixes an actual bug, and solves the original poster's problem.

Index: Compiler.java
===================================================================
--- Compiler.java      (revision 453)
+++ Compiler.java      (working copy)
@@ -211,16 +211,20 @@
       List commandObjcopy;
       
       commandObjcopy = new ArrayList(baseCommandObjcopy);
-      commandObjcopy.add(2, "srec");
+      commandObjcopy.add(2, "ihex");
+      commandObjcopy.set(3, "-j");
       commandObjcopy.add(".eeprom");
+      commandObjcopy.add("--set-section-flags=.eeprom=alloc,load");
+      commandObjcopy.add("--change-section-lma");
+      commandObjcopy.add(".eeprom=0");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
-      commandObjcopy.add(buildPath + File.separator + sketch.name + ".rom");
+      commandObjcopy.add(buildPath + File.separator + sketch.name + ".eep");
       if (execAsynchronously(commandObjcopy) != 0)
         return false;

       commandObjcopy = new ArrayList(baseCommandObjcopy);
       commandObjcopy.add(2, "ihex");
-      commandObjcopy.add(".flash");
+      commandObjcopy.add(".eeprom");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".hex");
       if (execAsynchronously(commandObjcopy) != 0)

The missing part, requiring a fix to the bootloader, is uploading the .eep file.

Here’s a patch for the bootloader. Disclaimer: I don’t know exactly why this works, and I don’t know if this works with anything but an Atmega168. But with this patch in place, at least the bootloader does not put rubbish into the eeprom of my Atmega168:

Index: ATmegaBOOT_168.c
===================================================================
--- ATmegaBOOT_168.c      (revision 453)
+++ ATmegaBOOT_168.c      (working copy)
@@ -482,6 +482,7 @@
           }
           if (getch() == ' ') {
             if (flags.eeprom) {                            //Write to EEPROM one byte at a time
+                address.word = address.word << 1;              //address * 2 -> byte location
                 for(w=0;w<length.word;w++) {
 #ifdef __AVR_ATmega168__
                   while(EECR & (1<<EEPE));
@@ -627,11 +628,10 @@
           if (address.word>0x7FFF) flags.rampz = 1;            // No go with m256, FIXME
           else flags.rampz = 0;
 #endif
+          address.word = address.word << 1;              // address * 2 -> byte location
           if (getch() == 'E') flags.eeprom = 1;
-          else {
-            flags.eeprom = 0;
-            address.word = address.word << 1;              // address * 2 -> byte location
-          }
+          else flags.eeprom = 0;
+
           if (getch() == ' ') {                            // Command terminator
             putch(0x14);
             for (w=0;w < length.word;w++) {                    // Can handle odd and even lengths okay

PS: something else I noticed while hacking my way through the bootloader is that the Atmega168 is special-cased because ‘the current avr-libc eeprom functions do not support the ATmega168’. It seems that the current (newer?) version of avr-libc supports eeprom functions for the ATmega168 just fine, though the special-cased m168 workaround results in a smaller bootloader hex file than using the functions from avr/eeprom.h

ahoeben: thanks for doing the work on this. I'll try to incorporate the compilation patches - I'm going to be making some changes to that part of the code anyway. The bootloader modifications are a lower priority, since we've already burned the bootloader onto a lot of boards. Next time we change it, though, I'll definitely try to get this in there.

You're welcome, it was (mostly) fun to do. I agree that the compilation changes are the most important for now. With those in place, at least you don't get a wrong .hex file if you use the EEMEM directive. The only functionality missing is setting initial values; reading and writing from within the sketch works fine.

The sooner the bootloader gets fixed, the fewer Arduinos with a (minor) bug will be out there. On the other hand, I fully understand that the bootloader changes would need extensive testing, and the number of people that will encounter the bug is very low (given that the bug has been in the bootloader since ~2005).

Too bad the Pro and the Pro Mini are both shipping with updated bootloaders. but this bug was not fixed. One more generation of shipping Arduinos that can not properly write to the eeprom using the bootloader :-(