Splitting power from Ethernet Shield Power over Ethernet

I am looking to build an Arduino based display box.

10x 4 -digit 7-segment displays and maybe some stand-alone LEDs.I am using the Sparkfun serial 4-digit displays (SparkFun 7-Segment Serial Display - Red - COM-11441 - SparkFun Electronics). I am not yet sure whether I can drive them all off the 5v from the Arduino. I think I may be OK (2 boards currently drawing about 20mA, so hopefully around 100mA for all 10), but I would like to be safe.

Since the PoE board is specified to be able to provide about 1.6A at 5V I was wondering whether I could split this to power the Arduino and the displays seperately. That way I will not have all of the 5V power going through the Arduino 5V pin.

Any thoughts.

The PoE board on the Arduino Ethernet board or shield outputs 12V so the power regulator on the Arduino itself converts it to 5V. The Yun itself takes in only 5V, so it requires a 5V PoE module (from what i gather, i am still looking for more answers). So if my understanding on the Yun is correct, then it should be a lot easier to get more power for add-on peripherals (5V ones that is).