I've got a controller+LCD board here, originally from a washing machine, which I'm hoping to eventually put into a produce drier to control heater and fan. It's based on an ATMEGA649V, which AvrDude wants to call "atmega649".
I've successfully built Optiboot, although I had to disable EEPROM support since the AVR compiler shipped by Debian lacks the relevant library. I've added 4MHz support and tweaked the load address to be 0xfe00. I can load the resulting .hex onto the processor using a USBASP programmer.
The sense of the reset signal is inverted, so I've modified AvrDude such that the arduino/serial protocol drives DTR in the opposite sense. At that point I can talk to the bootloader using either USBASP or Arduino protocol and can verify that memory below 0xfe00 is unprogrammed.
With the help of Juraj, I've installed the various target-specific files into ~/Arduino, and can build and load sketches into the board. I've fitted an LED onto B5 and it flashes as expected: three times when the bootloader starts up, and twice every second if the test script is running.
When only the bootloader's in Flash I get a triple-flash roughly every second: my understanding is that that's what's expected since when it tries to transfer control to a sketch it rolls all the way up memory until it's back in the bootloader and repeats.
If the test sketch is in Flash I get a double-flash every second which is the expected behaviour.
My problem is that once the test sketch is in Flash I can't reset back to the bootloader: either using the serial connection of by grounding the pin. As soon as I release the reset there's roughly a second's pause and then I get a double-flash: never the expected triple-flash.
As part of sorting out the reset polarity and clock speed I wrote a simple program that tried multiple port speeds and DTR polarities, issuing a synchronise command for each and looking for a response. This works fine with an Uno in all cases, or with this board provided that a sketch isn't loaded; however if a sketch is loaded it sees no response which appears to confirm that the bootloader is never being run.
Hfuse is 0xd6, I suspect it should be 0xd4 but in practice I've tried all values of the bottom three bits with no obvious change. This is, obviously, a bit of a pain to fiddle with since I have to connect the USBASP programmer to change the setting followed by a serial converter to test it.
Does anybody have any thoughts as to what my silly mistake is?