Certainly, the sketch seems to work fine if left alone and switching the output character shouldn't matter. My problem only appears when I push the Arcade Button (or I also tried a standard SPST switch). Pushing the button rapidly such as might occur during a video game seems to hang the "circuit" even though there is nothing in the sketch that references Pin 8 or a button / switch on that pin. I have tried both both pulling the pin up to 5V and pulling it down to ground and both can cause the "circuit" to hang.
If you were reffering to my circuit, then yes, the USB Ground is indeed connected to the bread board's ground rail. The black wire heading out the left is a USB cable that was cut and stripped then inserted into the breadboard.
Also if I recal, even if I remove the orange jumper in my circuit, the "circuit" can hang. This all leads me to believe that it's a momentary voltage drop causing either the Arduino to hang or the USB communications to fail and leaves the Arduino waiting for the USB bus to become free. As you mentioned USB is very sensitive to voltage... But honestly I am just guessing without an oscilloscope to check these theories.
It may also just be that the breadboard is not a good circuit for data lines and that I have to build a proper circuit on a PCB.