Putting "#define SERIAL_ISR" at the top of a PDE file would stop the Arduino library from grabbing the interrupt.
Moving the interrupt definition to the header would mean some variables contained in wiring_serial would need to be made global... specifically rx_buffer, rx_buffer_head, and rx_buffer_tail. These could all be put into a single struct to not pollute the namespace too much.
The other option I see is to make the interrupt call another function via a function pointer, and allow that function pointer to be redefined(it would default to the normal serial receive routine). Using a function pointer might cause some performance problems...
It sounds to me like you are ready to "graduate" from the Arduino development environment to a full-fledged C/C++ environment in which nothing is hidden or packaged in third-party libraries.
Yes, it is tempting to fix it with suggestions like yours, but at some point this feature creep would get in the way of what Arduino is supposed to be: simple. There are all sorts of places in the code where things can be customized, pointerized, #define'd, etc. but I think it's best left simple.
EDIT: Ah yes...here is the quote I was looking for:
"Every program in development at MIT expands until it can read mail."
Ya, I already have Eclipse setup with the AVR plugin (which is pretty kick ass). I also have a decent Xcode project setup.
The main reason I want to use the Arduino IDE is because I want to be able to distribute the program... hex files aren't very friendly. I probably just need to bundle avrdude and have a nice GUI to upload the hex file.
Hey, that's right. That code compiles because wiring_serial.o isn't linked because nothing in that object file is accessed. I haven't tested it, but it should work as long as no built-in serial functions are ever called. I was using Serial.begin()... but it's easy enough to just roll my own.