Hi,
The good news is... yes, of course, you can. The not-so-good news is... no, you can't just throw a switch with its resistor on A4-A5 if you intend to use I2C for something else.
This article: http://hackaday.com/2016/11/07/diy-i2c-devices-with-attiny85/ explains how to make a slave I2C device with an ATTiny85, such as a joystick, buttons and so on.
Now, a LED screen is much more responsive on SPI and, IMHO, single buttons are a waste of analog pins.