fastest way to right shift an array of 4 bytes

Casting the byte array to an unsigned long may not work as you expect or may not be portable because of endian issues.
This may or may not be a problem for you.