A possibly easier solution is to take a different port expander.
The PCA9505 offers 40 I/O lines, and even has 100kΩ pull-up resistors built in making connecting buttons really convenient. I2C communication, single interrupt line (ports can be masked so you only get interrupts from the interesting ones), 8 I2C addresses so 8x40=200 I/O possible without I2C extenders. They're pretty affordable, listed at USD 3.75 on Digikey.
Even more I/O offered by the CY8C9560A chip. This one has 60 I/O ports and all 7 bits of the I2C address can be set giving a total of 127 possible addresses! As each chip needs 2 I2C addresses you're limited to 63 of these chips on a single bus, still a total of 3,780 IO ports available. That ought to be enough 
USD 7.13 on Digikey.
Note that both these chips are available as SMD packages only so you'll need a breakout board.