A Xiao is pretty small; how small do you need?
At some point, a board's size is dominated by connectors (arguably, the Xiao is already there, though presumably you can get smaller by implementing even fewer pins.
There's the AVR64dd14, supported by MegaTinyCore. SOIC14, more code space and 8k of RAM, hardware support for I2C, and cheaper than a tiny85.