Another thing to try is moving the chip to another spot on the breadboard or another breadboard.
If have experienced bad contacts on one mine, althoug not shortycuts.
I was finally able to upload the Blink sketch to the ATtiny2313! I tried Ernis suggestion and moved it to a different breadboard and it worked right away. I feel kind of stupid for not thinking of trying that on my own.
Thanks so much for your help, guys.