The USBtinyISP includes 1.5K series resistors which will limit the current when there is a wiring problem but will also create a voltage divider with things like an LED + resistor. If you use a programmer that does not include series resistors (or the resistors are a lower value), you will be able to have a wider variety of things connected to the ISP pins. For example, I routinely use an Uno as a programmer and have never had a problem with LEDs on the ISP pins.