Ok, so in order to turn on an LED situated at the intersection of the layer X and column Y, you need to write 1 into the Xth bit of the anode shift register and 1 into the appropriate bit of the appropriate cathode shift register. The main idea is that both the anode and cathode shift registers need to receive a 1 bit in order to turn on the LED. That's nice. It seems that I misunderstood it earlier, but now everything is clear.
I'm not so worried at the moment about how exactly I write those bits into the shift registers, as that seems to be working. On the other hand I'm very much worried about the fact that turning a layer on also turns on 1 or 2 other layers below and/or above it, but not as brightly as the targeted layer. Also, if I turn off all the layers, the two middle layers turn on at full brightness. That's very worrying... seems like a hardware bug...
For testing purposes I have disabled the whole multiplexing and everything that can make debugging hard. My whole loop() function is empty now, I just test things in the setup() function.
For example, if my setup() contains this:
void setup ()
pinMode (PIN_MOSI, OUTPUT);
pinMode (PIN_SCK, OUTPUT);
pinMode (PIN_SS, OUTPUT);
digitalWrite (PIN_SS, LOW); //start transferring data
SPI.transfer (B00000000); //turn off cathode columns 8-15
SPI.transfer (B00000001); //turn on cathode column 0, turn off cathode columns 1-7
SPI.transfer (B00000001); //turn on the first anode plane (anode plane 0)
digitalWrite (PIN_SS, HIGH); //done transferring data
The code cannot get simpler than this. There's clearly no error in it and yet anode planes 1 and 2 also turn on in a dim manner next to anode plane 0 which turns on at full brightness, as instructed.
If in the last SPI.transfer(), which sets the anode planes, I transfer all 0 bits, the two middle anode planes (1 and 2) turn on.
Remember that I have a 4x4x4 cube driven by 3 daisy chained shift registers. The first 4 outputs of the first shift register on the line control the 4 anode layers through the MOSFETS and the next two shift registers on the line use all their 8 outputs to control the 16 cathode columns.
At the moment I don't have the slightest clue, what can be wrong, but it smells like a hardware problem to me...
Let me know if you suspect anything.