With serial, my program works the way I want it; without serial, it doesn't

Common ground means that all the grounds are connected to each other. That is there is a connection between all the points you call ground in your system.