On the Uno
D8 -_> Port B, bit 0
D9 --> Port B, bit 1
D10 --> Port B, bit 2
D11 --> Port B, bit 3
D12 --> Port B, bit 4
D13 --> Port B, bit 5
So there is a nice correspondence between pin # and port B bit #, the -8
If you look at the mega, it is different
D8 -_> Port H, bit 5
D9 --> Port H, bit 6
D10 --> Port B, bit 4
D11 --> Port B, bit 5
D12 --> Port B, bit 6
D13 --> Port B, bit 7
And in your code, you are using pins 12, 10, 11 so
12 - 8 = bit 4 of port B (Not correct, should be 6)
10 - 8 = bit 2 of port B (should be bit 4)
11 - 8 = bit 3 of port B (should be bit 5)
They are all off by 2 so you could edit your code to use a "magic offset" of -6 vs. -8
PORTB &=~(3UL<<(data_R1-6)); // data_R2 is LOW; data_R1 is LOW;
PORTB &=~(1UL<<(clockPin-6)); // digitalWrite(clockPin,LOW);
PORTB |= !((buffer[index]>>(7-i)) & 0x01) << (data_R1-6); // top set of rows
PORTB |= !((buffer[index+128]>>(7-i)) & 0x01) << (data_R2-6); // bottom set of rows
PORTB |= 1<<(clockPin-6);
But now you have Mega specific code.