I did some tests with compiling for ESP8266 using Arduino IDE 1.8.9 on the RPi. I definitely see it hitting 100% and it's quite slow, but it never hung the RPi. There were a couple compilations that were pretty slow and I started working on other things then came back later to check on it, so I don't know exactly how long it took. I did have one that ended with a couple errors shown in the Arduino IDE console window:
virtual memory exhausted: Cannot allocate memory
Then other times it compiled fine with no error and without too ridiculous of a delay (compiling for ESP8266 is always pretty slow, even on my more capable computer).
I know there were problems with caching of the compiled core library with previous versions of the Arduino IDE and ESP8266 core for Arduino. It may be that the caching process is problematic, and that was not apparent before because it wasn't being cached.
I just used some of the example sketches for the ESP8266WiFi library for compilation, since those are fairly large. I don't know whether the specific sketch you were compiling was part of the issue. Did you try compiling the minimal File > New sketch when you were using Arduino IDE 1.8.9? If so, did the same problem occur?