Sorry, but clearly you don't have much experience with electronics. That's okay, but you need to start taking account... Also you are not reading very closely, as I already commented on the wall adapter ground (there isn't one).
How do you suppose ground currents from the Due, would reach the other two boards? Now do you see?