- Compile libraries and core code. - done once, creates say "arduino.a", which gets linked.
This the the most serious "error" in your understanding. The libraries and core code needs to be recompiled each time you switch destination CPU type or clock speed. The IDE brute-forces this by recompiling libraries each time you change board types. and ALWAYS recompiles the core code each time you compile a sketch. I guess if you only have one arduino, only doing it once will last you quite a while...