First, thanks for the good feedback. It's good to get opinions from people who are pushing the limits of what the Arduino software can do, as it encourages us to get more features working.
We originally, in fact, controlled the upload process with an external makefile, but it had a lot of problems. Some versions of Windows required forward slashes as the path separator, some required double backslashes. We had issues with the working directory of the make command. We had difficulty getting error messages back from the commands launched by the makefile. Etc, etc.
In any case, we're trying to offer as much flexibility as possible in the compilation and upload processes through the use of the preferences.txt file.
Being able to use Arduino with the full power of C is also a good goal. Currently, you can use extra C files by creating extra tabs in a sketch and naming them with a C extension. They should be able to use the full range of C syntax (including union, etc). There's currently no way to prevent linking of the Arduino core, though, but it's a good idea.
In the future we're going to try to support multiple platforms, configurations, etc. It would be a good time to make sure that standalone C programs work too. You're right, it would be really nice to compile and burn the bootloader from within the IDE.
In summary, we're working on it.
As to how the backend works, right now Arduino will compile and link everything in the ARDUINO/lib/targets/arduino directory with each of your sketches. Each sketch gets the WProgram.h file included automatically, so you can add additional #include's there.
There are a bunch of options in the preferences.txt file (see the FAQ for directions on finding the preferences.txt file) for controlling the compilation and upload process. For example, you can set your clock speed, mcu, upload rate, whether or not to erase before uploading, etc.