The chip documentation reads "corresponding port pin is internally pulled up with a 100 k? resistor" when you set the register 0x0C (not sure why you have 0x12) to "1". In your case "all" due to the 0xFF.
Nick's Example Shows :
expanderWriteBoth (GPPUA, 0xFF); // pull-up resistor for switch - both ports
...where GPPUA is defined as : #define GPPUA 0x0C
So maybe that is part of the issue?
To me, 100K is a damn weak pull up... so weak in fact that I would tend to ignore it even exists.