So, after going throug the display driver datasheet, I tried various things these days, capacitors included. I somehow can reduce the noise or whatever it is that's causing the issues with AnalogRead, but can't get it completely right.
I can reduce the noise to an almost bearable level by using a 330µF cap across the OLED input, reducing the oled contrast (SETCONTRAST command), reducing Pre-charge (SETPRECHARGE command) and slowing the refresh rate considerably (SETDISPLAYCLOCKDIV command).
If I dim the display and slow it down to a visible flicker, I get an almost stable AnalogRead, with a painful display to watch.
I hooked an old analog oscilloscope to vcc/gnd and these are the results. In this case a Nano runs on 3.3v powered from a laptop USB.
Wave without Oled connected, running sketch:
Wave with Oled connected, running sketch with no hardware/software noise reducing measures:
There's some weird stuff going on here.