But all that would take too many cycles and sample-rate would lower too much. To be honest, I could just add Envelope at 8-bits, but it would require multiplication or division and it slows the whole thing down to a point that I get like 1 voice.
Now, imagine the current Beat707 SY LE, which does 8 voices, now, combine that with 4 PWM outputs, there's no extra cycles added, and the Envelope math is very simple and is not per-sample calculated, so it would still fit on the code. Doing the VCA outside in analog domain would make all this work like a charm. Its only 4 voices, but still, 4 dual OSC voices, not bad for a single ATmega328. 
Another path would go nuts and use another chip for processing audio, like my XMOS Shield idea, but that's another talk, check the other thread.
Wk