Like the other poster said - too much current is needed.
Hook the boards up to an external power supply that's able to power it all. It's a good test to ensure this is the problem!
If you don't want a power adapter in your design, consider using multiple USB plugs - like the old USB hard-disk's had.
I noticed there's one common resister for all the segments in the display. How have you avoided a change in LED brightness as the number of segments illuminated changes?