I've also written code where I had to drop all use of pinMode and replace with writes to the registers, so that it doesn't have to include pinMode(), saving the 170 bytes that took.
It's more than 5x slower, too. It's not just the interrupt safety, it's the lookup to convert pin numbers to port and bit. Whereas writing a register is really fast