I've only had experience with smaller OLED's however from your symptoms it sounds like the display is drawing slugs of current. I can't see why.
However I suggest you get a 100µF capacitor (or 47 to 500 µF). Connect it across the OLED power input. Hopefully the capacitor can supply the slugs (transients) of current without having anything else effected.
BTW 9V batteries (the rectangular ones usually used in a smoke alarm) are very anemic and don't have very much current capacity. However the USB should be able to power your display.