Indeed, throttling down the clock would be my suggestion and this will likely affect millis(), since that runs off a Timer and the Timer in turn runs off (a derivative of) the core clock. But maybe this is already accounted for in the core you use. See also here: RP2040 set clock speed - #2 by PaulRB (no conclusive answer though, just a suggestion of what you could try).
If you adjust the clock and millis() breaks, you'd have to adjust the Timer settings that millis() uses. It's not a big modification, but it'll require some rooting around in the core files.