Mega startup from external power

I have mega hooked up to a bunch of LED matrixes. The system recieves data wirelessly and saves it to a memory card. Every minute or so the arduino reads from the memory card and updates the led matrixes. Everything works perfectly when hooked up to the usb port of my computer. When I unplug the usb and power from an external source, everything still works. However if I unplug the external power and plug it back in, sometimes the mega does not initialize and just sits there or all of the matrixes show yellow. I know this can be caused by the bootloader receiving random data at startup and causing the arduino to not initialize. Is there a way to stop the serial data for the first few seconds while it boots up? Remember I do have to have the xbee always plugged in. Thanks in advance. For a little more info on my setup check here.

I would check the rise time on the power supply, with so much current draw I suspect the the voltage rise is too slow to trigger the reset circuit in the arduino chip. It could be you need a power supply that can supply a little more current.

I've got a 6V 2.5A power source hooked up. I don't think that is the problem.

Since I'm using a Mega, I decided to use Serial1 to send my data from the xbee, however it doesn't seem to be sending any data through. I should just be able to change Serial.begin(9600); to Serial1.begin(9600); in my setup if(Serial.available()) {Code} to if(Serial1.available()) {Code} in my loop and then connect the Xbee RX to Mega TX1 and the Xbee TX to Mega RX1.

Do I have to do anything else to get the extra Serial ports to work?

Got it working, sometimes you just have to unplug some wires and plug them back in. Frustrating. I'll post some updated images of the project here shortly.