serial OLED on Mni Pro serial port

I have a 4dsystems serial oled display on the RX & TX lines of the mini pro connected to the sparkfun USB programming board. When I program the mini and leave it attached the display will not light. When I unplug the USB board the display works fine. I'm guessing its because the USB board and oled board are both using the one serial port. I put two 1K resistors on the RX & TX lines of the oled display but still nothing until I unplug the programming board.

Any easy fixes besides a sotfware serial option?

Thanks,
MIke