Think I found the solution. Very simple in actuality. I didn't use udp.listen() as it was only putting packets out. I think it was outputing packets from the same port that it was broadcasting to meaning that it was receiving and sending packets from the same port. I added a udp.listen() with a port different from the one I was using and it seemed to fix my problem.
That's awesome. Motto of this story is never give up. Not something I would of thought would be a problem.
I wish that library had some comments or a readme on how it works.