If you're convinced it's nothing to do with using the Serial pins for column addressing then take the Serial calls out of the combined code and see what it does.
But I'd also check the wiring. I'm not sure the columns are connected where you think they are.
Steve