PJON communication stops working

I'm using the PJON communication library in SoftwareBitBang mode. I have a "controller" node and all the external nodes have a communication wire that comes into a single PIN in the controller. Many of the nodes seem to work consistently pretty quickly. But some of the more complex nodes seems to lose communication ability or it takes them many retries (and maybe 10 or 20 seconds) before they are successful. The "complex" nodes, for example, have multiple RFID readers hanging off them. It seems that as soon as I add "looking for a tag on the reader" into the loop, the communications start to degrade. It mostly appears to be the sending that stops working.
For example:

Controller sends a "start" communication to node 1.
Node 1 receives the communication and sends back a "okay I'm starting" message.
That works just fine.
Now node 1 starts watching the RFID readers on it. It finds a tag and tries to send a message back to the controller to say "Hey I found this tag" and at this point it tries, and tries and tries and MAYBE after 20 or 30 attempts it finally gets the message through to the controller.

The RFID readers are a good 4 or 5 feet away from the communications wires.
I'm kinda at a loss. I've done a lot of software massaging to make sure there are not gaps when the comm system is not being called. For example I've written code such that when it tries to say "hey I found a tag" that is ONLY does communications for 5 or 6 seconds, so it's NOT even looking at the RFIDs during that time, and that doesn't seem to help at all.

I am currently using just a twisted pair of wires from CAT5 cables as the comm/gnd wires, with a simple female jumper cable soldered onto the end and plugged into the Arduino. I'm wondering if trying something like shielded coaxial would be worth it or if there is just something else going on here. If I just put a node that all it does it look for comms and respond then it seems to work just fine, so there is something in all the OTHER stuff going on in the node that makes comms degrade.

Any thoughts would be appreciated. Oh, i'm using mostly Megas. The "controller" is an UNO.