I'm happy I can answer my own question. 
Turns out the flash memory can be corrupted from low voltage and the simple solution is to enable the brown out detection via the fuses. Like this if the voltage drops below 2.7v (for my setup) the ATtiny85 will shut down and restart when voltage goes back to safe level.
You can see it on page 144 of the datasheet.
Here's the fuse settings I use, you can see under "HIGH fuse presets" header the setting for brownout detection.