From what I've seen and read the R4 Minima and R4 WiFi are using an internal oscillator on the Renesas RA4M1 called HOCO and neither board has any sign of an external resonator/crystal. I'd expect that to be subpar compared to an external quartz crystal.
I did some testing to compare a few boards and the Minima is far better than the WiFi and better than the UNO R3! These are the clocks relative to a Raspberry Pi 4 (running chrony over wired connection) with a basic linear correction.
The four boards are all connected to my Pi 4 over USB. It's made me ponder where the improvement could be coming from and USB is one potential source. I did find Renesas KB: Can the embedded USB run without an external oscillator on RA Family MCU Groups? which perhaps implies this is possible. That would leave the question as to why the WiFi can't do this too. From schematic and a few docs, it looks like the ESP32-S3 is (normally) handling the USB on the WiFi variant so maybe this is robbing it of the ability to do this clock improvement?
I'm not very familiar with this hardware so would be good to get some more informed views on what's going on here.