i don't understand why there would be a latency, other than when adjusting a controler, the time needed for the sample rate to be synchronised with the new information from the button you're turning.
other than that, it's like the blipping of the little led on the arduino board : if i set it to a desired frequency, like 100 ms off , 200 ms on , 50 ms off etc etc ... the difference is that it's in midi info, not led light. and each note of the synth/keyboard is a different led i'm playing with, one by one :
500 ms this note, with 90 of velocity, then
500 ms this other note, with 90 of velocity, then
500 ms this other note, with 120 of velocity , etc etc ...
But maybe i'm wrong somewhere ?