OK, I hope this can help figure it out.
Part of the problem, at least with the Note II, is timing. The error I get when debugging the USB host code is a timeout (error code 0xFF) on one of the USB packets. This doesn't happen if (for instance) I output a whole lot of serial data to slow the sketch down. It is not enough to add a delay() in the main loop, since the whole thing fails if we delay() too much as well. Adding a delay(50) between each call to Usb.Task() and a few delay(100) at strategic places inside USBHost::Task() makes it work pretty reliably. I haven't had the time to make it work well yet or I would post code.
Another issue is the fact that the USB code stops working if you soft-reset the Due, such as by opening the serial console after a sketch has started running. This can be worked around by uploading the sketch, then unplugging the Due and as fast as humanly possible after plugging it in, you open the debug console. I guess this works because you can actually soft-reset the Due before the USB initialization takes place, and therefore it doesn't try to initialize it twice.
If you have the phone connected to the Due and your sketch either crashes or your code is timed wrong, you might confuse the USB stack on the Android. This is only solved by rebooting. So if it stops working, you might get it running again by simply rebooting the Android. I power-cycle the Due between every run.
I ran a modified version of the ADK2012 sketch and library, compiled with the ADK2012 version of the IDE. This powered up the USB port even when running on external power, so there's nothing wrong with the hardware. There has to be a mistake somewhere in the library, I just don't have enough knowledge to find it!