rheine:
I want to know if I can use the ICSP header to connect to my attiny85 dev board that has a 5X2 ICSP to program it.
No.
Most of the pins on the ICSP header would be OK but one of them is RESET and you can't control that in software.
rheine:
Or is this not what it is for? Is there a guide for this?
It's an input, not an output.
You can use it to program the Arduino directly instead of using the serial port. You can use it to burn new serial bootloaders, etc., or even bypass the bootloader altogether.