Can you please explain why the ATmega328p hardware I2C would prevent me from doing what I described above?
Because the TWI hardware in the ATmega328p, in slave mode, checks for the slave address itself, you write the address to react on into the TWAR register. You can use the TWAMR register to mask some or all bits of the address. This way you're able to react on some ranges or even all addresses sent to the bus but you don't get access to the sent address, so you cannot behave differently depending on the address sent.
You can use an ATtiny with a USI interface because there the software checks for the address and reacts accordingly, the hardware there is basically only a special shift register.