I hope I can explain this well, but if swapping columns is something you need to do a lot of, create a 720 element array of pointers to 4 element arrays. Then you just need to swap pointers instead of elements.
(You might also create a 4 element struct to efficiently store your 1 int and 3 bytes and use pointers to these structs instead of arrays. you would lose the convenience of standard array notation.)