Tomppu, the thread you linked to actually contains the explanation and the solution to the flickering problem you encountered. Start at this post to see what I mean. The solution is to output the BAM waveform followed by a mirror image of it.
I don't have time to modify the code right now, but I might later this week. If you can get a working version using MIBAM (mirror image bit-angle modulation) I would be glad to test it!
Edit: There is a dedicated MIBAM thread here. After reading through, on the Atmega328 at 16MHz, I think it is possible to drive 64 channels at 120Hz with 11-bit resolution (2048 levels). At 10-bits you could drive 128 channels, etc. Then again that is based on assembly performance so I'm not sure what the difference would be using the Arduino IDE. But it's encouraging!