Pulse reset low to start the bootloader (this is done by the DTR line of the serial adapter chip and a cap, google dtr reset circuit)
It communicates via STK500 protocol - I think there's a doc from Atmel that describes it.
Running from command line, you can run avrdude in a very verbose mode (see the docs for avrdude for details on how to enable this); the maximum verbosity mode will print every character sent or received to the console. Once you get close from the above documents, you can use that to see what you're doing wrong.