The avr-gcc compiler supports the ATmega64M1, so it should be possible.
The Arduino IDE uses the file 'boards.txt' for the definition of the boards with the microcontrollers.
On my computer it is located at: ....arduino-1.0.5/hardware/arduino/boards.txt
Try to understand how the fuses and settings are for the original Nano.
The bootloader and the fuses are probably the easiest part.
The arduino integration will require a "core" which sets up the registers and hardware. You will also have to have a pin map stored under the variants/ folder to map the pins to the pin names/numbers. Finally, the boards.txt entry has to be defined for the compiler and programming.
Some of the functionality may work with the default core and pin map, but you will likely have some analog pins and PWM pins that do not function.
I made a ATMega128a core and pin map. You will need to read up on the datasheet to complete this.
thanks a lot for the really good answer and the nice introduction of the use of Arduino.
I just look through the different files and found out, that it is a little bit complicated for me.
How can i put the new informations about the µC in the optiboot files? What do i have to change?
How can i find out the different information about the Atmega64M1 for the boards.txt. Is there someone how can give me this change from the datasheets?