If you post your code (in code tags!) it would be helpful.
Code tags are created by clicking on the code button in in the editor's menu.
There are generally two simple means to smoothen jittering days:
- Averaging over a number of measurements
- Quantizing the data (reducing the resolution)
Averaging is done by summoning up a certain number of measurements and deviding the result by the number of measurements.
Quantizing can be done by mapping the data from e.g. values between 0 and 4095 to 0 .. 1023 (y = x/4), just to name a common method.
Be aware that the ESP32 lacks accuracy in certain areas of ADC.
There are quite a number of webpages regarding this topic, just as a reference
https://forum.arduino.cc/t/fixing-the-non-linear-adc-of-an-esp32/699190
.https://esp32.com/viewtopic.php?t=2881&start=30
https://www.esp32.com/viewtopic.php?t=19804
https://docs.espressif.com/projects/esp-idf/en/v4.4.3/esp32/api-reference/peripherals/adc.html
and many more.
The expressif page lists also a possibility to reduce noise by adding a capacitor...
I guess posting the code might be a helpful step. You have already the attention of skilled members 
Die to the special time of the year I'm not sure that I can assist you in the next days but there are plenty of supportive people around here!
Good luck!
BTW: I would not put much effort in the non linear part of the ESP32 ADC curve but just cut off these data. It will only reduce the usable movements of the stick close to the left, right, forward or backward edges. Smooth and repeatable values around the center should be much more relevant...