EEMEM directive and EEPROM storage in .HEX files

I can have a look if you point me at where the buildproces is in the IDE sources... Like with the makefile, I won't really know what I'll be doing, but I can poke at the code until it works :wink:

The patch above needs some work; it does not like it when there is no .eep file. Is there such a thing as an "elseif" statement in a makefile?

update:
/app/Compiler.java and /app/AvrdudeUploader.java look like the most likely suspects. Time to brush up my java...

Looks like I'll have a patch next week. The changes are very minor, but I need to do some testing with real projects before I can vouch for the code...

Hmm, it seems that my initial enthusiasm was somewhat premature... It seems to work fine, until I declare an array in EEMEM, at which point the whole eeprom writing stage fails. I could use an extra pair of eyes...

Index: AvrdudeUploader.java

===================================================================

--- AvrdudeUploader.java      (revision 453)

+++ AvrdudeUploader.java      (working copy)

@@ -44,6 +44,7 @@

     } else {
       Collection params = getProgrammerCommands(Preferences.get("upload.using"));
       params.add("-Uflash:w:" + buildPath + File.separator + className + ".hex:i");
+      params.add("-Ueeprom:w:" + buildPath + File.separator + className + ".eep:i");
       return avrdude(params);
     }
   }
@@ -62,6 +63,7 @@

       "-b" + Preferences.getInteger("boards." + Preferences.get("board") + ".upload.speed"));
     commandDownloader.add("-D"); // don't erase
     commandDownloader.add("-Uflash:w:" + buildPath + File.separator + className + ".hex:i");
+    commandDownloader.add("-Ueeprom:w:" + buildPath + File.separator + className + ".eep:i");
 
     if (Preferences.get("boards." + Preferences.get("board") + ".upload.disable_flushing") == null ||
         Preferences.getBoolean("boards." + Preferences.get("board") + ".upload.disable_flushing") == false) {
Index: Compiler.java

===================================================================

--- Compiler.java      (revision 453)

+++ Compiler.java      (working copy)

@@ -211,16 +211,20 @@

       List commandObjcopy;
       
       commandObjcopy = new ArrayList(baseCommandObjcopy);
-      commandObjcopy.add(2, "srec");
+      commandObjcopy.add(2, "ihex");
+      commandObjcopy.set(3, "-j");
       commandObjcopy.add(".eeprom");
+      commandObjcopy.add("--set-section-flags=.eeprom=alloc,load");
+      commandObjcopy.add("--change-section-lma");
+      commandObjcopy.add(".eeprom=0");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
-      commandObjcopy.add(buildPath + File.separator + sketch.name + ".rom");
+      commandObjcopy.add(buildPath + File.separator + sketch.name + ".eep");
       if (execAsynchronously(commandObjcopy) != 0)
         return false;
 
       commandObjcopy = new ArrayList(baseCommandObjcopy);
       commandObjcopy.add(2, "ihex");
-      commandObjcopy.add(".flash");
+      commandObjcopy.add(".eeprom");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".hex");
       if (execAsynchronously(commandObjcopy) != 0)

The patch above changes the following:

  • Compiler.java is changed so it creates an .eep file in ihex format, instead of a .rom file in srec format
  • Compiler.java is changed so the .hex file does not contain the eeprom section
  • AvrdudeUploader.java is changed so it also uploads the .eep file to eeprom space

You can use this sketch to test the modifications:

#include <avr/eeprom.h>

byte EEMEM NonVolatileChar = 123;
int  EEMEM NonVolatileInt = 12345;
//char EEMEM NonVolatileString[15]; = "Hello world"; 

void setup() { 
    char SRAMchar;
    int SRAMint;
    char SRAMstring[15];

    SRAMchar = eeprom_read_byte(&NonVolatileChar);
    SRAMint  = eeprom_read_word(&NonVolatileInt);
    //eeprom_read_block((void*)&SRAMstring, (const void*)&NonVolatileString, 15); 

    Serial.begin(9600);
    Serial.println(SRAMchar,DEC);
    Serial.println(SRAMint,DEC);
    Serial.println(SRAMstring);
}

void loop() {
  
}

When used like this, the code works fine; The NonVolatileChar and NonVolatileInt are read as expected. Uncommenting the EEMEM declaration of NonVolatileString breaks the eeprom upload process for me, and not even the NonVolatileChar and NonVolatileInt read back correctly anymore. This is the same with both the IDE changes in this post and the makefile changes I posted before, though the makefile by default does not verify the written data.

Update: To my untrained eye, it looks like the bootloader is putting bytes in the wrong places in the eeprom section. Something about byte vs word boundaries and/or big vs little endian-ness. The bootloader is special-casing the Atmega 168 when it comes to writing the eeprom section, and I don't understand that code. I'll have to build me an AVR ISP so I can see what exactly gets put in the eeprom, and to see if it works if I use the AVR ISP instead of the bootloader.

Finally had the time to construct my usbtinyisp. Unfortunately, the bootloader seems to foobar the eeprom data during upload. The eeprom data seems to come across intact when I choose to use the usbtinyisp for programming the sketch, but the bootloader munges up a couple of bytes per handful of bytes.

So even if I could get this fixed, chances of it getting in are very slim, right? Seeing that it would need a bootloader update...

Either way, it would be great if the following patch could get in; it fixes an actual bug, and solves the original poster's problem.

Index: Compiler.java
===================================================================
--- Compiler.java      (revision 453)
+++ Compiler.java      (working copy)
@@ -211,16 +211,20 @@
       List commandObjcopy;
       
       commandObjcopy = new ArrayList(baseCommandObjcopy);
-      commandObjcopy.add(2, "srec");
+      commandObjcopy.add(2, "ihex");
+      commandObjcopy.set(3, "-j");
       commandObjcopy.add(".eeprom");
+      commandObjcopy.add("--set-section-flags=.eeprom=alloc,load");
+      commandObjcopy.add("--change-section-lma");
+      commandObjcopy.add(".eeprom=0");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
-      commandObjcopy.add(buildPath + File.separator + sketch.name + ".rom");
+      commandObjcopy.add(buildPath + File.separator + sketch.name + ".eep");
       if (execAsynchronously(commandObjcopy) != 0)
         return false;
 
       commandObjcopy = new ArrayList(baseCommandObjcopy);
       commandObjcopy.add(2, "ihex");
-      commandObjcopy.add(".flash");
+      commandObjcopy.add(".eeprom");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".hex");
       if (execAsynchronously(commandObjcopy) != 0)

The missing part, requiring a fix to the bootloader, is uploading the .eep file.

Here's a patch for the bootloader. Disclaimer: I don't know exactly why this works, and I don't know if this works with anything but an Atmega168. But with this patch in place, at least the bootloader does not put rubbish into the eeprom of my Atmega168:

Index: ATmegaBOOT_168.c
===================================================================
--- ATmegaBOOT_168.c      (revision 453)
+++ ATmegaBOOT_168.c      (working copy)
@@ -482,6 +482,7 @@
           }
           if (getch() == ' ') {
             if (flags.eeprom) {                            //Write to EEPROM one byte at a time
+                address.word = address.word << 1;              //address * 2 -> byte location
                 for(w=0;w<length.word;w++) {
 #ifdef __AVR_ATmega168__
                   while(EECR & (1<<EEPE));
@@ -627,11 +628,10 @@
           if (address.word>0x7FFF) flags.rampz = 1;            // No go with m256, FIXME
           else flags.rampz = 0;
 #endif
+          address.word = address.word << 1;              // address * 2 -> byte location
           if (getch() == 'E') flags.eeprom = 1;
-          else {
-            flags.eeprom = 0;
-            address.word = address.word << 1;              // address * 2 -> byte location
-          }
+          else flags.eeprom = 0;
+
           if (getch() == ' ') {                            // Command terminator
             putch(0x14);
             for (w=0;w < length.word;w++) {                    // Can handle odd and even lengths okay

PS: something else I noticed while hacking my way through the bootloader is that the Atmega168 is special-cased because 'the current avr-libc eeprom functions do not support the ATmega168'. It seems that the current (newer?) version of avr-libc supports eeprom functions for the ATmega168 just fine, though the special-cased m168 workaround results in a smaller bootloader hex file than using the functions from avr/eeprom.h

ahoeben: thanks for doing the work on this. I'll try to incorporate the compilation patches - I'm going to be making some changes to that part of the code anyway. The bootloader modifications are a lower priority, since we've already burned the bootloader onto a lot of boards. Next time we change it, though, I'll definitely try to get this in there.

You're welcome, it was (mostly) fun to do. I agree that the compilation changes are the most important for now. With those in place, at least you don't get a wrong .hex file if you use the EEMEM directive. The only functionality missing is setting initial values; reading and writing from within the sketch works fine.

The sooner the bootloader gets fixed, the fewer Arduinos with a (minor) bug will be out there. On the other hand, I fully understand that the bootloader changes would need extensive testing, and the number of people that will encounter the bug is very low (given that the bug has been in the bootloader since ~2005).

Too bad the Pro and the Pro Mini are both shipping with updated bootloaders. but this bug was not fixed. One more generation of shipping Arduinos that can not properly write to the eeprom using the bootloader :frowning:

Great to see a fix turning up in SVN. Yay!

Ladyada fixed the bootloader side of things for the Atmega328 chips she's currently sending out, so Atmega328-based boards will be among the first that eventually also support uploading to the EEPROM area.

For now, at least we can use the EEMEM directive, without Arduino creating .hex files that are too big.

If you can figure out a way to get avr-objcopy to create the .eep files without generating a warning if they're empty, that would be even better. Otherwise, you'll probably need to extract them manually on the command line.

Sorry I did not catch that before. This seems to work for me:

--- /app/Compiler.java      Mon Oct 13 20:04:37 2008
+++ /app/Compiler.java      Mon Oct 13 20:02:31 2008
@@ -215,6 +215,7 @@
       commandObjcopy.set(3, "-j");
       commandObjcopy.add(".eeprom");
       commandObjcopy.add("--set-section-flags=.eeprom=alloc,load");
+      commandObjcopy.add("--no-change-warnings");
       commandObjcopy.add("--change-section-lma");
       commandObjcopy.add(".eeprom=0");
       commandObjcopy.add(buildPath + File.separator + sketch.name + ".elf");

Edit: This is (obviously) against revision 513, not 514.

Great, that seems to work. I'm generating the .eep files again. Do you think it makes sense to upload the EEPROM data with avrdude on every upload?

Trying to upload anything to the eeprom section on a non-fixed bootloader always results in an upload error (unless you want to turn off verification). So at the very least uploading the .eep section should probably be an option that lives in boards.txt.

Come to think of it, that could be a very good solution, because for example all 328s that Ladyada is shipping currently have the fix in place, so they are eeprom-capable. Atmega328-based Arduinos would have a different entry in boards.txt. And if a user wants to prevent his/her eeprom section from getting overwritten, he/she can edit the boards.txt file to disable the upload. One could even make two copies of a board config; one with and one without eeprom uploads.

Sounds like a good plan. I'll add it (at some point). We will be switching to the updated bootloader as well.

Can somebody summarize the status of this thread? Is the EEMEM directive now supported and .eep upload working with the bootloader?

Since the 0013 release and newer (or was it 0012?), .eep files are created, but not uploaded. You can upload them manually using avrdude.

should I create a patch that allows manual upload of .eep files out of the IDE?
Can they be uploaded by the bootloader or only via isp programmer?
Or maybe the IDE must upload the .eep file if the size is not zero or at least ask the user in this case?

Maybe users want to read the eeprom and have a look at the contents in the Editor as well?

Or is the eeprom a topic for professional users only and they all know how to handle avrdude?

Here's a patch that should let you specify in boards.txt whether or not the eep file gets uploaded.

Index: AvrdudeUploader.java
===================================================================
--- AvrdudeUploader.java      (revision 540)
+++ AvrdudeUploader.java      (working copy)
@@ -49,6 +49,10 @@
     } else {
       Collection params = getProgrammerCommands(uploadUsing);
       params.add("-Uflash:w:" + buildPath + File.separator + className + ".hex:i");
+      String eepFile = buildPath + File.separator + className + ".eep";
+      Boolean uploadEEPROM = Preferences.getBoolean("boards." + Preferences.get("board") + ".upload.eeprom_section");
+      if (uploadEEPROM && (new File(eepFile)).exists())
+        params.add("-Ueeprom:w:" + eepFile + ":i");
       return avrdude(params);
     }
   }
@@ -67,6 +71,10 @@
       "-b" + Preferences.getInteger("boards." + Preferences.get("board") + ".upload.speed"));
     commandDownloader.add("-D"); // don't erase
     commandDownloader.add("-Uflash:w:" + buildPath + File.separator + className + ".hex:i");
+    String eepFile = buildPath + File.separator + className + ".eep";
+    Boolean uploadEEPROM = Preferences.getBoolean("boards." + Preferences.get("board") + ".upload.eeprom_section");
+    if (uploadEEPROM && (new File(eepFile)).exists())
+      commandDownloader.add("-Ueeprom:w:" + eepFile + ":i");
 
     if (Preferences.get("boards." + Preferences.get("board") + ".upload.disable_flushing") == null ||
         Preferences.getBoolean("boards." + Preferences.get("board") + ".upload.disable_flushing") == false) {

The reason not to turn this option on for all boards is that there is a bug in most 168-based bootloaders out there that will make the eep uploading process fail. This bug was fixed with the 328-compatible bootloader recently, so all 328-based arduinos should be fine.

ok, thank you.

I will test it later and combine it with my local AvrdudeUploader modifications.