I seem to recall that problem analysis mention on the forum in the past, but for the life of me I can't think of the solution or what to search on. If you have another arduino board laying around you might consider loading it with the arduinoiSP sketch and wiring it to your target board and see if it suffers the same or not.