I think it should be possible, even if you only set a daily alarm on the RTC, so that it fires an interrupt at exactly the same time every day.
Then the tiny would just need to set the same time as the RTC alarm time whenever it receives the interrupt. No polling needed.
For reasonable accuracy during the day, the tiny couldn't really run on it's internal oscillator though, or the timing could go a fair way out between interrupts, over a 24 hour period.
That would be the minimalist approach - very little ATtiny code overhead, but not particularly accurate.
You could set more frequent alarms - more accuracy at the cost of higher code requirements.
Quite doable though, I reckon. And with a good crystal and the right load capacitances, even a daily time update could produce reasonably accurate results.
Until I saw your post, I wasn't even aware that the "Time" library could operate without an RTC, but a quick test showed that it works fine. Never too old to learn.
Edit: Of course, you'd need to use a UNO or something to initially set the RTC time and alarm(s), then connect it to the ATtiny.