A single object can be used for multiple displays, here is a discussion about using five displays with a shared buffer
https://forum.arduino.cc/t/u8g2-oled-multiplexing-with-tca9548a/1058114
A few notes on that code:
It uses U8g2 instead of U8glib.
begin() is called for each display. U8g2 does not allocate the buffer in begin(), that is done from the constructor. begin() performs any needed initialization of the display controller, clears the display, and turns the display on.
The constructor is wrong, it specifies software I2C when it should be hardware I2C (this was the solution for the problem the OP was having).
The sketch uses a full buffer, a page buffer will use less ram and will use the firstPage()/nextPage() loop similar to U8glib.