ATMEGA2560 in my own PCB design will not execute code, but code uploads fine.

Other Hardware 3rd Party Boards
1

A few months ago, I decided to design my own PCB. I am making a flight computer for my model rockets, and my sketches were growing to be enormous in size. The current version of my flight computer using an Arduino Nano has something like 99% of its space used. I wanted to make a really small version of the Arduino Mega so I could have a crap ton of space to work with. I basically copied the official Arduino schematics except I left out the USB part of it, as well as a lot of pins. I used Seeedstudio to make the PCBs.

Surprisingly, the PCB seems to work fine, the only problems I have encountered are due to me being a novice solderer with those tiny pins. Anyway, I successfully soldered the ATmega2560 to my PCB. I am using an USBasp programmer to program it. I successfully installed the bootloader onto the ATmega2560, and the Arduino IDE has no problem uploading code to it successfully using USBasp.

The problem that I am having is the fact that no matter what I upload to it, the ATmega2560 refuses to execute the code. I do not understand why. Even the simple blink program does not work. I have tried everything that I can think of. Still nothing. Reset is pulled high. Power is 5v. Lots of decoupling caps. Latest version of all software and firmwares. I have been doing some research, and it may be the fuses on the ATmega2560. I am not sure though, any help would be much appreciated!

Be warned, the following was made in Eagle by someone who has yet to master it. I also think I used too much of the autorouter for the board, but it ain't bad for a novice.

Schematic: Here
Board with everything soldered to it: Here

Avrdude when I upload the code:

         System wide configuration file is "C:\Program Files (x86)\Arduino\hardware\tools\avr/etc/avrdude.conf"

         Using Port                    : usb
         Using Programmer              : usbasp
         AVR Part                      : ATmega2560
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    10     8    0 no       4096    8      0  9000  9000 0x00 0x00
           flash         65    10   256    0 yes    262144  256   1024  4500  4500 0x00 0x00
           lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : usbasp
         Description     : USBasp, http://www.fischl.de/usbasp/

avrdude: auto set sck period (because given equals null)
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0x1e9801
avrdude: NOTE: "flash" memory has been specified, an erase cycle will be performed
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: auto set sck period (because given equals null)
avrdude: reading input file "C:\Users\JOHNSL~1\AppData\Local\Temp\build9025638294211856625.tmp/Blink.cpp.hex"
avrdude: writing flash (2322 bytes):

Writing | ################################################## | 100% 1.28s

avrdude: 2322 bytes of flash written
avrdude: verifying flash memory against C:\Users\JOHNSL~1\AppData\Local\Temp\build9025638294211856625.tmp/Blink.cpp.hex:
avrdude: load data flash data from input file C:\Users\JOHNSL~1\AppData\Local\Temp\build9025638294211856625.tmp/Blink.cpp.hex:
avrdude: input file C:\Users\JOHNSL~1\AppData\Local\Temp\build9025638294211856625.tmp/Blink.cpp.hex contains 2322 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.66s

avrdude: verifying ...
avrdude: 2322 bytes of flash verified

avrdude done.  Thank you.

Fuse Positions:

>avrdude -C ../etc/avrdude.conf -c usbasp -b 19200 -p m2560 -v

avrdude: Version 6.0.1, compiled on Apr 15 2015 at 19:59:58
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "../etc/avrdude.conf"

         Using Port                    : usb
         Using Programmer              : usbasp
         Overriding Baud Rate          : 19200
         AVR Part                      : ATmega2560
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    10     8    0 no       4096    8      0  9000  9000 0x00 0x00
           flash         65    10   256    0 yes    262144  256   1024  4500  4500 0x00 0x00
           lfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  9000  9000 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : usbasp
         Description     : USBasp, http://www.fischl.de/usbasp/

avrdude: auto set sck period (because given equals null)
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e9801
avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as FD

avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as FD
avrdude: safemode: Fuses OK (H:FD, E:D8, L:FF)

avrdude done.  Thank you.

Thanks for any help you are able to provide!

2

Your fuses are ok.

My guess is that you have an usbasp with an old firmware that does not handle addressing over 128KB correctly. The bootloader should be located near the end of 256K, older firmware versions write it near the end of 128K, when verifying they read it back in from there so the verification succeeded, leaving you with the impression everything went ok.

You can check whether this assumption is correct by having usbasp read the bootloader back in into a hex file and check if the location of the bootloader is correct.

Towards the end of the hex file you'll see tons of 0xff's followed by the actual bootloader code. This region must be above (3*64K). So in the hexfile this region must be preceded by a so called extended linear address record, a line like this:

:020000040003F7

02   : size
0000 : address, ignored
04   : record type, 4==extended linear address
0003 : means for all subsequent data records (type 0), add 0x30000 to their address.
F7   : a checksum

The start of the bootloader code will look like this:

:20E000000D9489F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F109

20   : size: 32 bytes
E000 : address
00   : record type, 0==data
0D9489F10D... : the data

Which means it starts at 0x3e000 so it can be 8KB in size.

If the usbasp firmware is to blame, you can upgrade it with e.g. the ArduinoISP sketch.
But then it makes more sense to first try to program the bootoader to the 2560 with the ArduinoISP sketch. If this is an option for you, get it from here, it is much more stable. You will need avrdude 6.1 to correctly address over 128KB.

3

PeterVH:
Your fuses are ok.

My guess is that you have an usbasp with an old firmware that does not handle addressing over 128KB correctly. The bootloader should be located near the end of 256K, older firmware versions write it near the end of 128K, when verifying they read it back in from there so the verification succeeded, leaving you with the impression everything went ok.

You can check whether this assumption is correct by having usbasp read the bootloader back in into a hex file and check if the location of the bootloader is correct.

Towards the end of the hex file you'll see tons of 0xff's followed by the actual bootloader code. This region must be above (3*64K). So in the hexfile this region must be preceded by a so called extended linear address record, a line like this:

:020000040003F7

02  : size
0000 : address, ignored
04  : record type, 4==extended linear address
0003 : means for all subsequent data records (type 0), add 0x30000 to their address.
F7  : a checksum





The start of the bootloader code will look like this:

:20E000000D9489F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F109

20  : size: 32 bytes
E000 : address
00  : record type, 0==data
0D9489F10D... : the data




Which means it starts at 0x3e000 so it can be 8KB in size.

If the usbasp firmware is to blame, you can upgrade it with e.g. the ArduinoISP sketch.
But then it makes more sense to first try to program the bootoader to the 2560 with the ArduinoISP sketch. If this is an option for you, get it from [here](https://raw.githubusercontent.com/PeterVH/Arduino/issue-3321/build/shared/examples/11.ArduinoISP/ArduinoISP/ArduinoISP.ino), it is much more stable. You will need avrdude 6.1 to correctly address over 128KB.

Alright, I dumped the flash and got this hex: Here

From what you told me, it appears that my blink sketch is on there at the beginning, and the bootloader is at the end. I am going to reinstall the bootloader with your version of ArduinoISP. I had already installed the bootloader with my USBasp, but I did that before I updated the firmware on the USBasp. I updated the USBasp with the latest firmware found here: USBasp - USB programmer for Atmel AVR controllers - fischl.de

The bootloader does not seem to be in the correct place, and, as far as I can tell, it does not have the same hex you gave me, which may be the end problem here. I will check back in after I install the bootloader with ArduinoISP.

Edit: I am also going to upgrade AVRDUDE to 6.1.

Edit 2: The hex file I pulled is only 131kb, which confirms your theory of the programmer not being able to access the full 256kb. When I get back to my computer I will do as you suggested earlier.

4

PeterVH:
Your fuses are ok.

My guess is that you have an usbasp with an old firmware that does not handle addressing over 128KB correctly. The bootloader should be located near the end of 256K, older firmware versions write it near the end of 128K, when verifying they read it back in from there so the verification succeeded, leaving you with the impression everything went ok.

You can check whether this assumption is correct by having usbasp read the bootloader back in into a hex file and check if the location of the bootloader is correct.

Towards the end of the hex file you'll see tons of 0xff's followed by the actual bootloader code. This region must be above (3*64K). So in the hexfile this region must be preceded by a so called extended linear address record, a line like this:

:020000040003F7

02  : size
0000 : address, ignored
04  : record type, 4==extended linear address
0003 : means for all subsequent data records (type 0), add 0x30000 to their address.
F7  : a checksum





The start of the bootloader code will look like this:

:20E000000D9489F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F10D94B2F109

20  : size: 32 bytes
E000 : address
00  : record type, 0==data
0D9489F10D... : the data




Which means it starts at 0x3e000 so it can be 8KB in size.

If the usbasp firmware is to blame, you can upgrade it with e.g. the ArduinoISP sketch.
But then it makes more sense to first try to program the bootoader to the 2560 with the ArduinoISP sketch. If this is an option for you, get it from [here](https://raw.githubusercontent.com/PeterVH/Arduino/issue-3321/build/shared/examples/11.ArduinoISP/ArduinoISP/ArduinoISP.ino), it is much more stable. You will need avrdude 6.1 to correctly address over 128KB.

Alright, I finally figured this thing out. You were right, the usbasp cannot access more than 128kb of memory. I did as you suggested and updated avrdude to 6.1. Then I used an Arduino Nano as an ISP and uploaded the bootloader to the ATmega2560. After that, I used the Nano to pull all the flash to verify that the bootloader was installed correctly. It matched what you said exactly. I then used to nano to upload the blink code and it worked! I then wanted to see what happened if I used my usbasp to upload the blink program. It uploaded fine, but the code did not run. It actually corrupted the bootloader and I had to use the Nano to reinstall it. Is there a usbasp that is not limited to 128kb? Anyway, thanks for your help in getting this figured out.

5

SchrodingersCat_:
Is there a usbasp that is not limited to 128kb?

Yes, PeterVH explained that at the bottom of his post.

6

Did you install this version: usbasp.2011-05-28.tar.gz?
Inside the change log it says the version is 1.4 and it mentions:

  • added support for controllers with flash >128kb (by Slawomir Fraś)

I have no usbasp, so I don't know whether this works. (I am curious about this though, info on the www on this matter is a bit contradictory).

Edit: just remembered you already said you upraded to the latest version from the usbasp site.
So the conclusion is that it does not work?

7

PeterVH:
Did you install this version: usbasp.2011-05-28.tar.gz?
Inside the change log it says the version is 1.4 and it mentions:

  • added support for controllers with flash >128kb (by Slawomir Fraś)

I have no usbasp, so I don't know whether this works. (I am curious about this though, info on the www on this matter is a bit contradictory).

Edit: just remembered you already said you upraded to the latest version from the usbasp site.
So the conclusion is that it does not work?

I will try to reinstall it and see if it works. When I tried to program using the usbasp, it corrupted the bootloader on the ATmega2560 and nothing would work. I will flash the usbasp again and see what happens.

8

PeterVH:
Did you install this version: usbasp.2011-05-28.tar.gz?
Inside the change log it says the version is 1.4 and it mentions:

  • added support for controllers with flash >128kb (by Slawomir Fraś)

I have no usbasp, so I don't know whether this works. (I am curious about this though, info on the www on this matter is a bit contradictory).

Edit: just remembered you already said you upraded to the latest version from the usbasp site.
So the conclusion is that it does not work?

Alright, I have been working with my usbasp. This is the one I have here.

I reflashed the latest firmware (usbasp.2011-05-28.tar.gz) onto the usbasp using avrdude 6.1 and your version of ArduinoISP. It can access all 256kb of flash now as it did when I flashed it earlier, but there is a new problem with it. It writes the bootloader in two places when I use it to burn the bootloader to the ATmega2560.

It writes it at 0x1e000 and 0x3e000. I have no idea why. It is probably a flaw in the usbasp firmware. I did check to see if the firmware was loaded to the usbasp correctly, and it was. The hex matched.

This is what the usbasp with the latest firmware wrote to the ATmega2560: here. It makes no sense, but with the new firmware on the usbasp, it can access all 256kb. I think what is happening here is that the usbasp is seeing the 256kb as 2x 128kb chunks and thinking it needs to write the bootloader to both chunks. That is my theory at least. The bootloader is correctly written using a standard ArduinoISP setup.

Here is the info I pulled from the usbasp.

Microsoft Windows [Version 10.0.10240]
(c) 2015 Microsoft Corporation. All rights reserved.

>avrdude -C ../etc/avrdude.conf -c avrisp -P COM6 -b 19200 -p m8 -v

avrdude: Version 6.1, compiled on Sep 27 2015 at 16:35:31
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "../etc/avrdude.conf"

         Using Port                    : COM6
         Using Programmer              : avrisp
         Overriding Baud Rate          : 19200
         AVR Part                      : ATmega8
         Chip Erase delay              : 10000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom         4    20   128    0 no        512    4      0  9000  9000 0xff 0xff
           flash         33    10    64    0 yes      8192   64    128  4500  4500 0xff 0x00
           lfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           calibration    0     0     0    0 no          4    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : STK500
         Description     : Atmel AVR ISP
         Hardware Version: 2
         Firmware Version: 1.18
         Topcard         : Unknown
         Vtarget         : 0.0 V
         Varef           : 0.0 V
         Oscillator      : Off
         SCK period      : 0.1 us

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.04s

avrdude: Device signature = 0x1e9307
avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D9

avrdude: safemode: lfuse reads as FF
avrdude: safemode: hfuse reads as D9
avrdude: safemode: Fuses OK (E:FF, H:D9, L:FF)

avrdude done.  Thank you.

This is proving to be a tough one. I would like to get the usbasp working because it is so convenient to just plug into the ICSP header.

EDIT: One interesting thing I discovered is that just plugging in the usbasp to read the bootloader corrupts it. No writing, just reading. It is adding another copy of the bootloader at the end of the first 128kb. I don't think this is an issue that can be solved without screwing with the firmware, which is not something I have any skill in. It looks like I will be using a Nano to do all the programming. :frowning:

Thanks for the help guys!

9

I think another Arduino makes the best ISP programmer. Especially Leonardo-type Arduinos because with those you don't need to disable auto reset to use as ISP programmers. You can run the Arduino as ISP sketch on them, and it is easy to modify/customize the sketch and update. And I have yet to run into problems. These dedicated programmers such as usbasp and usbtinyisp and others are just small Arduinos with hex files for firmware instead of sketches. If you look at the schematics you can see they are just Atmel processors of some type with minimal support circuitry. Well, that is what an Arduino is. Yes, you can get the source code and customize or make corrections, enhancements, etc. for ATmega8 or ATtiny85-based programmers, but it is kind of a bother when you just want something that works. How about getting a Sparkfun Pro Micro or DF Robot Beetle (which are both just super-small Leonardos), and glue a 2x3 female header right to the bottom and wire it up permanently as your dedicated ISP programmer?

10

dmjlambert:
I think another Arduino makes the best ISP programmer. Especially Leonardo-type Arduinos because with those you don't need to disable auto reset to use as ISP programmers. You can run the Arduino as ISP sketch on them, and it is easy to modify/customize the sketch and update. And I have yet to run into problems. These dedicated programmers such as usbasp and usbtinyisp and others are just small Arduinos with hex files for firmware instead of sketches. If you look at the schematics you can see they are just Atmel processors of some type with minimal support circuitry. Well, that is what an Arduino is. Yes, you can get the source code and customize or make corrections, enhancements, etc. for ATmega8 or ATtiny85-based programmers, but it is kind of a bother when you just want something that works. How about getting a Sparkfun Pro Micro or DF Robot Beetle (which are both just super-small Leonardos), and glue a 2x3 female header right to the bottom and wire it up permanently as your dedicated ISP programmer?

I already have a bunch of Arduino Nanos I bought on eBay. I will probably just dedicate one as a permanent programmer. At any rate, it is still exciting that my custom designed PCB actually works! I think in the next version of it, I will stick a USB interface on it so I don't have to worry about all this insanity.

I have learned a lot from this experience. I have learned more about avrdude, bootloaders, and compiling from source than I ever have in the past. It has given me a lot to think about for my next flight computer design.

11

@SchrodingersCat_ : In another thread it is reported some usbasp's out there do correctly program the bootloader to a 2560. We try to figure out which ones. Yours clearly didn't, can you provide us a link of the programmer you have?

12

Hi,

Have you tried this?
I've had the same problem with a custom ATmega 2560 PCB, sketches uploaded without errors but the mega2560 would not execute the code. (However it didn't work with an Arduino as ISP either)
After I changed the high fuses to 0xD9 with the code from the link everything works flawlessly with the USBasp. No bootloader needed and it's much faster too :slight_smile:

Cheers

13

PeterVH:
@anon83577697_ : In another thread it is reported some usbasp's out there do correctly program the bootloader to a 2560. We try to figure out which ones. Yours clearly didn't, can you provide us a link of the programmer you have?

This one is the one I have.

I am going to try chillman's solution and see if that works.

chillman:
Hi,

Have you tried this?
I've had the same problem with a custom ATmega 2560 PCB, sketches uploaded without errors but the mega2560 would not execute the code. (However it didn't work with an Arduino as ISP either)
After I changed the high fuses to 0xD9 with the code from the link everything works flawlessly with the USBasp. No bootloader needed and it's much faster too :slight_smile:

Cheers

I will give it a shot and report back if it works or not. Thanks for the advice!

14

chillman:
Hi,

Have you tried this?
I've had the same problem with a custom ATmega 2560 PCB, sketches uploaded without errors but the mega2560 would not execute the code. (However it didn't work with an Arduino as ISP either)
After I changed the high fuses to 0xD9 with the code from the link everything works flawlessly with the USBasp. No bootloader needed and it's much faster too :slight_smile:

Cheers

Holy crap, that worked like a charm! I can't believe how simple the solution ended up being. Here I am, chasing down a firmware fault, when it was just a fuse that needed to be changed! Thanks for the help man! I burned the bootloader successfully and uploaded the blink sketch and it worked perfectly. It executes code like it should now. This has been a lot of work, but I'm glad we finally got this sorted out.

15

After you upload the bootloader with the new high fuse setting does the bootloader still work for serial upload after you have changed the fuses or only upload using programmer? After I set the high fuse to D9 and Burn Bootloader to ATmega2560 with my USBasp serial upload no longer works but Upload Using Programmer starts working. With high fuse D8 serial upload works but Upload Using Programmer never runs the program.

16

pert:
After you upload the bootloader with the new high fuse setting does the bootloader still work for serial upload after you have changed the fuses or only upload using programmer? After I set the high fuse to D9 and Burn Bootloader to ATmega2560 with my USBasp serial upload no longer works but Upload Using Programmer starts working. With high fuse D8 serial upload works but Upload Using Programmer never runs the program.

When I burn the bootloader with D9 and my usbasp, it burns it correctly. However, if I upload a program (by using "upload using programmer") with D9 and usbasp, it will wipe the flash entirely, including the bootloader, and write the program hex only. The program then runs fine without the bootloader. So it seems you don't need the bootloader with the D9 fuse setting if you are using a usbasp. I am using a custom PCB so I cannot comment on serial functionality as it is currently not in place on my custom PCB (check my first post for my design). But since it is wiping the bootloader when I go to upload a program, you may try burning the bootloader with D9, then changing it to D8 after you burn it but before you upload a program and then see if it works.

17

After I set the high fuse to D9 and Burn Bootloader to ATmega2560 with my USBasp serial upload no longer works but Upload Using Programmer starts working.

That is completely normal:
D9 => start at 0 => bootloader (just before 256K) does not work anymore.

With high fuse D8 serial upload works but Upload Using Programmer never runs the program.

That the boot loader works is normal too:
D8 => start at bootload section.

That the program never (or only in some cases?) works is not completely clear to me. The program counter sometimes wraps around at 256K and in other conditions it does not? For my ArduinoISP sketch tests, I have done dozens of tests in which I programmed a binary of > 100K to my mega and it worked every time (I had high fuse at D8 because I was not aware of this problem) .

18

SchrodingersCat_:
try burning the bootloader with D9, then changing it to D8 after you burn it but before you upload a program and then see if it works.

I tried that by burning the bootloader with high fuse D9 with the standard avrdude command used by the Arduino IDE:

avrdude -C"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\tools\avr/etc/avrdude.conf" -v -patmega2560 -cusbasp -Pusb -e -Ulock:w:0x3F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xD9:m -Ulfuse:w:0xFF:m 
avrdude -C"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\tools\avr/etc/avrdude.conf" -v -patmega2560 -cusbasp -Pusb -Uflash:w:"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\arduino\avr/bootloaders/stk500v2/stk500boot_v2_mega2560.hex":i -Ulock:w:0x0F:m

Then I changed the fuse to D8 with this command:

avrdude -C"C:\Program Files (x86)\arduino-1.6.5-r5\hardware\tools\avr/etc/avrdude.conf" -v -patmega2560 -cusbasp -Pusb -Ulock:w:0x0F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xD8:m -Ulfuse:w:0xFF:m

All commands completed successfully. Then I changed the high fuse setting in boards.txt to D8 and restarted the IDE and tried a serial upload with the same result as always: "avrdude: stk500v2_ReceiveMessage(): timeout"

PeterVH:
That is completely normal:
D9 => start at 0 => bootloader (just before 256K) does not work anymore.

Here's the strange thing though. If I burn the bootloader with high fuse set to D9 with my Atmel AVRISP mkII then I can do serial upload and Upload Using Programmer with the USBasp also works. After burning bootloader this way the LED blinks with the blink code built into the bootloader but when I burn bootloader with high fuse D9 with USBasp the LED doesn't blink even though with D8 it does. I've tried this with multiple Mega 2560s and multiple different versions of USBasp with the same result. So this tells me that high fuse D9 is not incompatible with a working stk500boot_v2_mega2560.hex bootloader, it's some difference in avrdude using USBasp

19

Then I changed the high fuse setting in boards.txt to D8 and restarted the IDE and tried a serial upload with the same result as always: "avrdude: stk500v2_ReceiveMessage(): timeout"

So usbasp loaded bootloader + hfuse D8 does not work. This is consistent with what SchroedingersCat_ reported. I stay with my hypothesis that usbasp did not correctly burn the bootloader. I ordered an usbasp so I can investigate this further. I expect the answer to be in the sources of the latest Fishel fw. There must be something wrong with the support for flash above 128K. I hope to make a fix such that this use case works correctly. Will report back if I find something...

Here's the strange thing though. If I burn the bootloader with high fuse set to D9 with my Atmel AVRISP mkII then I can do serial upload...

I can't explain that.
I repeated your experiment but with the ArduinoISP sketch as a programmer. From the moment I set the hfuse to D9, serial upload does not work any more. Setting it back to D8 makes it work again. But in my understanding, that is the expected behavior: D9 means start at 0x0000, so how could the cpu reach the bootloader?

...After burning bootloader this way the LED blinks with the blink code built into the bootloader

This does make sense: stk500boot_v2_mega2560.hex contains a blink sketch at address 0x0 and a bootloader near the end of flash. So if you set hfuse to D9, the blink sketch will run.

but when I burn bootloader with high fuse D9 with USBasp the LED doesn't blink even though with D8 it does.

As above, I expect this to be because the image that usbasp put in flash, is incorrect.

it's some difference in avrdude using USBasp

With that I totally agree.

20

PeterVH:
So usbasp loaded bootloader + hfuse D8 does not work.

If I burn bootloader with hfuse D8 using USBasp the bootloader does work fine but the Upload Using Programmer doesn't work. It's when I burn bootloader with hfuse D9 that it doesn't work. The reason it didn't work with hfuse D8 that time is because I burned the bootloader with hfuse D9 and then changed the fuse setting to D8 without reburning the bootloader as suggested in #15.

PeterVH:
I expect the answer to be in the sources of the latest Fishel fw.

My USBasp has the firmware it came from Ebay with. It gives the "avrdude: warning: cannot set sck period. please check for usbasp firmware update." message. I'm willing to flash any firmware to it in the interest of investigating this further. I'm using the version 6.0.1 of the avrdude included with the Arduino IDE(which is modified from the standard avrdude).

PeterVH:
I hope to make a fix such that this use case works correctly. Will report back if I find something...

Thanks so much for looking into this. I'd like to get a verification from someone else that has tried this as to whether my results are normal. There is more discussion of this issue here: Program doesn't run after Upload Using Programmer with USBasp (Mega2560) · Issue #246 · arduino/ArduinoCore-avr · GitHub. Testato, who recommended the change of hfuse to D9 in the other topic claims that serial upload should work fine with D9 and that I'm getting an unexpected result.