Thank you, I appreciate your recommendations! Finding all i-s to dot and t-s to cross to make sure this isn't a recurring theme is important to me 
The PC is powered by the same bench PSU (a second channel) because it' on-board on the robot that also has the motor controller.
There is some common ground (heh) and then a Y split to H-bridge vs LM350 regulator, after which there's a short cable (6") to the controller board. That Y splitter now has the big capacitor; it didn't before.
I did remove the 5V connector from my arduino-based programmer. I don't think I can do it for the others, because they use presence of 5V as a detector for an inserted cable. (The AVRISPmk2 also detects that RST can be pulled low, and if not, detects a reversed cable.) The USBtinyISP design has current limiting resistors on the programming pins already, but not on the 5V.