When I light up a single row the problem disappears.
I removed the UDN2981A for trouble shooting. The matrix code worked just fine 
Since the current I will be driving is below the 595 limits, I will leave the design like this (without using the UDN2981A).
Thanks for the suggestion.