The first attempt was using a 10 µF electrolytic. I still have not much of an idea really what I should be filtering exactly - what frequency/voltage/etc - it did stabilise the voltage enough to make the regulator work, but not perfectly stable.
I was looking also at this image from Wikipedia, which shows decoupling/bypassing as something any type can be used for:
The attempt with film caps (the 100 nF output is ceramic - got those as this value is needed all the time) didn't last long, just to see if it works, and it did. No longer time stability test with that. Maybe I should look into getting some more ceramic caps then. I got a box with like 30 values of film caps, something like that for ceramics would be nice 