When NOT to have a common ground?

No. It should be fine - in fact, in your case, common ground is necessary, unless you want to add in more components and a separate power supply. You only really need it if you're talking about, say, switching mains power.