How are the buttons connected to the 74HC165s?
If it works fine when you press single buttons that indicates the software is OK.
If it starts to go 'random' when you press several buttons, that sounds like you're short-circuiting the system or something. Measure the voltage at Vcc, does it droop?