ATMEGA2560 in my own PCB design will not execute code, but code uploads fine.

[SchrodingersCat_]

After you upload the bootloader with the new high fuse setting does the bootloader still work for serial upload after you have changed the fuses or only upload using programmer? After I set the high fuse to D9 and Burn Bootloader to ATmega2560 with my USBasp serial upload no longer works but Upload Using Programmer starts working. With high fuse D8 serial upload works but Upload Using Programmer never runs the program.

When I burn the bootloader with D9 and my usbasp, it burns it correctly. However, if I upload a program (by using "upload using programmer") with D9 and usbasp, it will wipe the flash entirely, including the bootloader, and write the program hex only. The program then runs fine without the bootloader. So it seems you don't need the bootloader with the D9 fuse setting if you are using a usbasp. I am using a custom PCB so I cannot comment on serial functionality as it is currently not in place on my custom PCB (check my first post for my design). But since it is wiping the bootloader when I go to upload a program, you may try burning the bootloader with D9, then changing it to D8 after you burn it but before you upload a program and then see if it works.

[PeterVH]

After I set the high fuse to D9 and Burn Bootloader to ATmega2560 with my USBasp serial upload no longer works but Upload Using Programmer starts working.

That is completely normal:
D9 => start at 0 => bootloader (just before 256K) does not work anymore.

With high fuse D8 serial upload works but Upload Using Programmer never runs the program.

That the boot loader works is normal too:
D8 => start at bootload section.

That the program never (or only in some cases?) works is not completely clear to me. The program counter sometimes wraps around at 256K and in other conditions it does not? For my ArduinoISP sketch tests, I have done dozens of tests in which I programmed a binary of > 100K to my mega and it worked every time (I had high fuse at D8 because I was not aware of this problem) .

[pert]

try burning the bootloader with D9, then changing it to D8 after you burn it but before you upload a program and then see if it works.

I tried that by burning the bootloader with high fuse D9 with the standard avrdude command used by the Arduino IDE:

Code: [Select]

avrdude -C"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\tools\avr/etc/avrdude.conf" -v -patmega2560 -cusbasp -Pusb -e -Ulock:w:0x3F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xD9:m -Ulfuse:w:0xFF:m
avrdude -C"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\tools\avr/etc/avrdude.conf" -v -patmega2560 -cusbasp -Pusb -Uflash:w:"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\arduino\avr/bootloaders/stk500v2/stk500boot_v2_mega2560.hex":i -Ulock:w:0x0F:m
Then I changed the fuse to D8 with this command:

Code: [Select]

avrdude -C"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\tools\avr/etc/avrdude.conf" -v -patmega2560 -cusbasp -Pusb -Ulock:w:0x0F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xD8:m -Ulfuse:w:0xFF:m
All commands completed successfully. Then I changed the high fuse setting in boards.txt to D8 and restarted the IDE and tried a serial upload with the same result as always: "avrdude: stk500v2_ReceiveMessage(): timeout"

That is completely normal:
D9 => start at 0 => bootloader (just before 256K) does not work anymore.

Here's the strange thing though. If I burn the bootloader with high fuse set to D9 with my Atmel AVRISP mkII then I can do serial upload and Upload Using Programmer with the USBasp also works. After burning bootloader this way  the LED blinks with the blink code built into the bootloader but when I burn bootloader with high fuse D9 with USBasp the LED doesn't blink even though with D8 it does. I've tried this with multiple Mega 2560s and multiple different versions of USBasp with the same result. So this tells me that high fuse D9 is not incompatible with a working stk500boot_v2_mega2560.hex bootloader, it's some difference in avrdude using USBasp

[PeterVH]

Then I changed the high fuse setting in boards.txt to D8 and restarted the IDE and tried a serial upload with the same result as always: "avrdude: stk500v2_ReceiveMessage(): timeout"

So usbasp loaded bootloader + hfuse D8 does not work. This is consistent with what SchroedingersCat_ reported. I stay with my hypothesis that usbasp did not correctly burn the bootloader. I ordered an usbasp so I can investigate this further. I expect the answer to be in the sources of the latest Fishel fw. There must be something wrong with the support for flash above 128K. I hope to make a fix such that this use case works correctly. Will report back if I find something...

Here's the strange thing though. If I burn the bootloader with high fuse set to D9 with my Atmel AVRISP mkII then I can do serial upload...

I can't explain that.
I repeated your experiment but with the ArduinoISP sketch as a programmer. From the moment I set the hfuse to D9, serial upload does not work any more. Setting it back to D8 makes it work again. But in my understanding, that is the expected behavior: D9 means start at 0x0000, so how could the cpu reach the bootloader?

 

...After burning bootloader this way  the LED blinks with the blink code built into the bootloader

This does make sense: stk500boot_v2_mega2560.hex contains a blink sketch at address 0x0 and a bootloader near the end of flash. So if you set hfuse to D9, the blink sketch will run.

but when I burn bootloader with high fuse D9 with USBasp the LED doesn't blink even though with D8 it does.

As above, I expect this to be because the image that usbasp put in flash, is incorrect.

it's some difference in avrdude using USBasp

With that I totally agree.

[pert]

So usbasp loaded bootloader + hfuse D8 does not work.

If I burn bootloader with hfuse D8 using USBasp the bootloader does work fine but the Upload Using Programmer doesn't work. It's when I burn bootloader with hfuse D9 that it doesn't work. The reason it didn't work with hfuse D8 that time is because I burned the bootloader with hfuse D9 and then changed the fuse setting to D8 without reburning the bootloader as suggested in #15.

I expect the answer to be in the sources of the latest Fishel fw.

My USBasp has the firmware it came from Ebay with. It gives the "avrdude: warning: cannot set sck period. please check for usbasp firmware update." message. I'm willing to flash any firmware to it in the interest of investigating this further. I'm using the version 6.0.1 of the avrdude included with the Arduino IDE(which is modified from the standard avrdude).

I hope to make a fix such that this use case works correctly. Will report back if I find something...

Thanks so much for looking into this. I'd like to get a verification from someone else that has tried this as to whether my results are normal. There is more discussion of this issue here: https://github.com/arduino/Arduino/issues/388. Testato, who recommended the change of hfuse to D9 in the other topic claims that serial upload should work fine with D9 and that I'm getting an unexpected result.

[PeterVH]

I did a few experiments.

After all the fact that upload using programmer (usbasp) combined with hfuse==0xD8 does not work is due to a small bug in the usbasp 1.4 fw (and it can't work at all with older versions).

I have written down my findings here

There is a fix over there too, with which switching between hfuse 0xD8 and 0xD9 is not needed.

@pert: I must admit I did not fully understand your question before doing some experiments myself. I think the link above answers your question.

[pert]

@PeterVH: Excellent investigation and explanation of what was causing the issue. I flashed your firmware to my USBasp and I can now Burn Bootloader and Upload Using Programmer with hfuse D8 and both work correctly. Thanks so much for your work on this!