After some more playing, way way too late at night, I have come to the conclusion that is odd problem must just be down to the 5v SPI levels causing reliability problems.
After finally getting the reader working again by connected both grounds, It decided to stop working and then to only work with both grounds disconnected!
I'm going to ditch the cheaper SD card modules and go for the micro SD reader with the proper level shifter. It's 4 times more expensive but at least I know it will keep working from one day to the next!
It's amazing that something that worked fine for months can suddenly decide to stop, but that's what comes of using things out of spec...
In the mean time, if anyone has a source of really economical 5V compatible SD card modules, in the UK, let me know. At the moment I'm looking at the one Hobby Components have for £4.50.