Hmmm... Have you tried a burn with SLOW_SCK on the USBasp? The only other thing that could be an issue is just the clock source. I still find it hard to believe that that crystal (+/-20 ppm typical) you have is worse accuracy than a resonator (+/-2000-5000ppm) that genuine Arduino's are produced all day long... have you tried a pair of 33pF caps in place of the 22pF crystal caps?
If you get a chance, please upload a picture of your breadboard setup.