UPDATE:
So I have another Attiny85 digispark board and I plugged the 5v output up to the attiny and it still doesn't work even with 5 volts. Yet it still works perfectly when plugged into the programmer.
Any idea why the attiny85 might totally refuse to work on power even from another attiny board?