The .ino file(s) of the sketch are compiled every time, but, when using the modern IDE versions, all other source files should be cached and only recompiled if they have been modified or you change to a different board.
I think the reasoning for always recompiling the sketch is that it was not worth the effort to set up caching for it, since the user will most often have made a change since the last compilation. I guess the one time that is not the case is when you have a failed upload, but hopefully that's not happening too often.
I recommend using the latest version of the Arduino IDE and making sure all your installed boards platforms are updated in Tools > Board > Boards Manager. The reason is that the caching system has improved over time, as well as the boards platforms configuration to enable the caching to work fully.
The ability to upload without recompile is provided by the official Arduino CLI tool. The
arduino-cli upload command only does an upload, meaning you must explicitly run the
arduino-cli compile command first to generate the binaries (though there is an
arduino-cli compile --uploadoption that provides the same combo procedure as the Arduino IDE.