Please provide a schematic. The mode set on the PCF8574 is a misnomer, it really only has an output mode where the output is low or a weak pullup for high. This is called quasi-bidirectional. In high mode the pin can be pulled low to indicate a low. As an output, low mode can sink about 20 ma, high mode can only source about 100 ua. All the details are in the data sheet.
When used with buttons typically the button is connected between the pin and ground. A high is written to the pin, which activates the weak pull up. The pin is then read and a low indicates a button press.