There is absolutely no connection between the 6V servo supply and the 5V and I2C on the Nano.
The grounds of the two power supplies must be connected together.
I agree. But the only possible connections are through the ground from the Nano to the controller (as per the AdaFruit wiring diagram) and/or through the house power and the wall warts. Those don't seem likely.
But hope arises! The sample program calls the Adafruit driver which in turn calls the Wire I2C library. I never read up on that library because I figured the manufacturer of the board knew how to use it. Well, I finally read up on Wire/I2C and found reference to pullup resistors. The Adafruit documentation makes no mention of those, so I looked to see if they were built into the controller board. Didn't seem to be. So I'll install them and see what happens.