I have something of the same problem.
I built an Atmega128 board.
I followed the Atmega128 Minimum Schematic - which seemed to generally follow the Atmega328 mini Pro.
I followed the previous Links directions posted above.
I connected an Arduino Mini Pro 328 via SPI to the Atmega128.
I can download code to the 128 via Programmer=ArduinoAsIsp.
Also when I connect the UsbToSerial (FTDI) to the PE0/RX0,PE1/RX1 , I can see/use the Serial from the program.
I can download (supposedly) the ATmegaBOOT_168_atmega128.hex to the Atm128.
But, if I again connect the UsbToSerial (FTDI) to the PE0/mosi,PE1/miso,Dtr->reset , change Programmer=AvrISP , then the Arduino just spits back NoSync error.
I have been back and forth repeatedly.
I can download code via ArduinoAsISP - but the FTDI to Atmega128 just gives me NoSync.
What am I missing ?