The ICSP connection is MOSI, MISO, SCK, VCC, GND, and RESET. Five out of six of those signals are on the ICSP connector on the USBasp. The RESET line on the ICSP connector is an out out pin from the processor to drive the RESET input of the target machine. It is not connected to the RESET line of the USBasp. You will need to locate the RESET pin of the ATmega8A processor on the USBasp. That's pin 29 counting counter-clockwise from the dot or the 4th pin clockwise. On my USBasp it appears to be connected to a empty jumper marked J2.
Readme - fischl.dewww.fischl.de/usbasp/Readme.txt
"J2 Jumper for firmware upgrade (not self-upgradable)
Set this jumper for flashing the ATMega(4)8 of USBasp with another working
I think that mean you can put a jumper (solder blob) across J2 and it connects the RESET output pin to the RESET line so you can use it for an input. With the jumper in place you can use an ISP connected to the ICSP connector to burn fresh firmware.
Thanks for the quick response!
So I basically just have to connect the pins 1, 2, 4, 7, 9 (MOSI, VCC, GND, SCK, MISO) from the 10 pin connector on the programmer to the appropiate pins on the Arduino (according to the Arduino ISP tutorial)? And I think on the USBasp clone I linked to you can see a black jumper. Maybe it is the one for the reset pin?