Can this damage the ATtiny85? What are your recommendations?
Unlikely.
You can try and if you are really concerned, put a small resistor (1k or 10k) to the supply rail and run some simple task on the mcu (like flipping some pins). The resistor there limits the current.
Or you can run it with just 2 - 3 batteries.