Code stops reacting to encoder interrupts at 92% of dynamic mem used.

I am using the same code with two displays: OLED SSD1306, 128x64 and LCD2004. With LCD, mem is at 70% and code flies. With oled, when mem is at 91%, it's still pretty fast, but with additional 1%, everything falls apart. It's basically unusable, as encoder turns are read with only about 5% success. I stuffed everything I could at PROGMEM, is there anything I can save mem space with? Is there a guide for this? I can't really post the code as I am not the author, I just modified it to meet my needs.

This is the thread in DIYaudio with link to original code: