The analog signal to the arduino input is not stable at all.
I assume you mean the Arduino's readings are not stable. If the source of the signal is not stable an Arduino cannot change it.
Do you have any idea to make it stable?
Do multiple readings and average them, be sure to use datatypes that can hold the sum. Also an often used practice is to do 2 measurements an ignore the first...
I tried Analog input smoothing with average value.. But...it is not working...
try - http://www.arduino.cc/playground/Main/Statistics
- or - http://arduino.cc/playground/Main/RunningAverage
I am also thinking about Rotary Encoder instead of Pot. What do you think about it?
I think nothing as I don't know what your goal is.