Excellent idea! A sawtooth or triangle wave and comparator will produce an excellent PWM signal for the LEDs. If you run the oscillator at well above audio ranges you shouldn't have to worry about the signal leaking into any audio lines.
Now that I think about it, the triangle wave is definitely better than a sawtooth. With a sawtooth you would have many LED's turning on (or off) at the same time. With a triangle wave, both the ON and OFF times will be staggered based on the pot settings.