Another concept that may be missed is the highest I2C address is 128 (7 bit) or 1024 (10 bit). As I recall 7 bit address is much more popular.
I love the fact you are planning to blow your capability wide open, by a factor of 100 (?), to 300k pins. I2C is slow, to read in all that data will take some time. I suggest a two-for-one change to your plan. Add a layer of modularity, rather than trying to get all the pins back to one Arduino, bring them back to several smaller Arduinos then use a higher speed bus to transfer from the small Arduinos to a larger one. If you use a Mega you can use Serial1, 2, and 3 and pull back three at a time using Serial TTL.
Without using the PCA9564, how many slaves can you get to work? How many slaves do you need? Divide the second by the first to get the number of “reader” Arduinos you would need.