@Grumpy_Mike
Yes, spot on, I totally forgot to update my schematics but I have 10 resistors on the columns and not on the 8 rows as shown in the image.
Also somehow I got my LED's working without the transistors and with a very good brightness. (LED's require 20mA). Does it harm anything if I continue without?
@KE7GKP
Haha yes I used all the pins 
I would have done it differently now but I guess that is what learning is all about.. I totally dropped my first idea. That idea was the result of a noob trying to figure out a creative way around
I think your multiplex is a really good idea so I'm gonna update the schematics with a 4051 between it.