Long runs of 0xFF in PROGMEM cause upload error

I've been hacking together a sketch to program a 24c16 eeprom to mend a broken Philips monitor, and tripped over a problem with data stored in program memory using PROGMEM.

I'm using Arduino 0016 (Windows XP) with a Diecimila board.

In summary, if the data contains a run of 152 or more bytes of 0xFF, you get an error like this when you upload to the board:

avrdude: verification error, first mismatch at byte 0x0088
  0xff != 0x01
avrdude: verification error; content mismatch

I tried two different Diecimila boards and both failed, but the address of the mismatch was slightly different - weird!

The simple sketch below illustrates the problem. Comment out the #define line to make it upload correctly. Note that even then it doesn't actually do anything: I've reduced the code to a relative minimum just to show the upload error.

#include <Wire.h>
#include <avr/pgmspace.h>

// Comment out the line below to remove the error
#define FAILDUDE  

const int deviceAddress = 0x50;

const byte firmware[] PROGMEM = { // Stored in flash memory
#ifndef FAILDUDE  
  // >=152 bytes of mixed data is OK
  0x01,0x10,0x00,0x18,0x05,0x30,0x00,0x19,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x08,0x00,0x0C,0x05,0x20,0x00,0x15,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x10,0x00,0x14,0x05,0x50,0x00,0x18,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x28,0x00,0x13,0x05,0x60,0x00,0x1D,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x00,0x00,0x08,0x05,0xC8,0x00,0x16,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x00,0xC1,0x00,0x11,0x05,0xC8,0x00,0x25,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x77,0x00,0x1F,0x06,0x40,0x00,0x19,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x07,0x00,0x19,0x05,0xB0,0x00,0x20,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x47,0x00,0x3C,0x05,0xF8,0x00,0x19,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0x01,0x37,0x00,0x1D,0x05,0xE0,0x00,0x1F//,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF
#else  
  // >= 152 bytes of 0xff gives error:
  // avrdude: verification error, first mismatch at byte 0x0088
  //       0xff != 0x01
  // avrdude: verification error; content mismatch
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,
  0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF//,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF
#endif  
};

// Code below doesn't matter, except that the firmware array must be referenced, 
// else it won't be linked in!

void setup(void) {
  Serial.begin(19200);
  Wire.begin();
}


void loop(void) {
  Serial.print("\nWriting");
  // We're writing 2048 bytes = 128, 16-byte pages
  unsigned int eeAddress;

  const byte *pbuf = firmware;
  for ( eeAddress = 0; eeAddress < 2048; eeAddress += 16) {
    i2c_eeprom_write_page(deviceAddress,eeAddress,pbuf);
    pbuf += 16;
    Serial.print(".");
  }
  Serial.print("\n");

  // loop indefinitely
  while(1) {
  }
}

// Write a 16-byte page: assumes data is in program memory
void i2c_eeprom_write_page( int deviceaddress, unsigned int eeaddress, const byte *data) {
  byte page = eeaddress >> 8;
  Wire.beginTransmission(deviceaddress+page);
  Wire.send((byte)(eeaddress & 0xFF));
  for ( int ib = 0; ib < 16; ib++) {
    char c = pgm_read_byte(data++);    
    Wire.send(c);
  }
  Wire.endTransmission();
  delay(5);
}

I fixed my problem by declaring the data as an array of ints instead of bytes: a waste of space but OK for this quick hack.

PS, can I respectfully suggest that the forum adinistrators try registering for this forum again themselves? You have to guess a unique user ID and display name with no feedback, and every time you get it wrong you have to enter everything again.

Heh....this is a neat problem.

The problem is that FLASH is programmed one "page" at a time (128 bytes per page I think). AVRDUDE optimizes the uploading process by not bothering to upload "blank" pages, and since 0xFF represents an unprogrammed FLASH byte, it doesn't bother uploading pages which are just full of 0xFF bytes.

Normally this is not a problem because AVRDUDE begins with an entire chip erase so that all bytes are set to 0xFF and uploading a page's worth of 0xFF's doesn't make sense and doesn't change the program. But the Arduino bootloader doesn't implement the chip erase command so the FLASH isn't erased, AVRDUDE doesn't send your page of 0xFF's, and the program fails to verify.

Some options I can see:

• Modify your program to not have so many 0xFF's, as you've already done
• Modify AVRDUDE to not do this blank-page check and just upload everything (see the file stk500.c at about line 816).
• Contact the AVRDUDE author and encourage him to add a command-line option to skip the blank-page check, then integrate this option into the Arduino IDE
• Use a bootloader that implements chip erase

Thanks for the clear explanation. I was puzzled why it took 152 bytes of 0xff to trigger the problem, but I guess this was enough to create a single page containing only 0xff.

I don't fully understand what you mean by:
"Normally this is not a problem because AVRDUDE begins with an entire chip erase so that all bytes are set to 0xFF and uploading a page's worth of 0xFF's doesn't make sense and doesn't change the program. But the Arduino bootloader doesn't implement the chip erase command ..."

Is it AVRDUDE of the bootloader that erases everything? (Or not in this case).

AVRDUDE tries to erase everything by sending a command to the bootloader ('chip erase'). The bootloader that comes with the Arduinos says "Thanks very much for the command, not gonna do it, wouldn't be prudent."

So AVRDUDE thinks the chip has been erased, all bytes are 0xFF, and proceeds accordingly.

If you're interested, the offending line is at line 501 of "ATmegaBOOT_168.c". This shows how the AVRDUDE command to erase the chip is ignored.

        /* R: Erase device, don't care as we will erase one page at a time anywa
y.  */
        else if(ch=='P' || ch=='R') {
                nothing_response();
        }