I missed your last reply, so let me back the conversation up a couple days first.
I think you might be assuming "Ground" is somehow special. It isn't, it's just a reference point. You can probe any two points in a circuit with a multimeter and get a measurement. Ground isn't any different in this regard, it's just that normally ground is the reference on which everything else is based. So, when you say something is "+5v", that's 5v above ground. Ground is only 0v because you're comparing it to itself, so the difference should very well be 0v.
Differential is a special case because it doesn't reference ground, it references another signal line. Electrically, it's still a measurement between two points, it's just that neither of those points are (necessarily) ground. Single-ended inputs imply that the reference is ground, and requires the system and input grounds to be tied together.
Again, in your case, the amp system and input grounds are tied together in two places: The PSU and the RCA input jacks. So you have a loop. It doesn't mean there's any appreciable current flowing through that loop, it just means there is one. That's not a problem in and of itself, and not something to be "fixed". The problem is when there's a difference in potential between those two ground points. The best way to fix that is to ensure the two ground points have low impedance paths back to the PSU ground. (Easier said than done sometimes.)
If you take away the RCA shield, you're causing the amp to reference the PSU ground as the audio ground (since they're connected internally anyway). The audio signal will then be referenced against a ground that is not at the exact same potential as the device creating the output (the Mighty), so there will be noise which is the difference between ground as the Mighty sees it and the (PSU) ground as the amp sees it. Using a signal ground (RCA shield) will attempt to force them (the grounds) to the same potential, at the risk of causing current to flow down the path of least resistance, which may alternate as the load changes.
Ceramic caps across the input will shunt noise, but it will do so as a high-pass filter with a high cutoff frequency. Typically, you would use, say, a 30pF cap. This dumps RF noise straight to ground, but noise throughout (and even well above) the audio band will not be affected. It can still be helpful if the audible noise is a lower harmonic, but that's a long shot.
The transformers help probably (not 100% sure -- comments welcomed) because the path through it becomes high impedance. Noise is going to take the easiest path to the point of lower potential (Ground again), which will probably be elsewhere.