Is this a known behavior for the 328?
Inputs not connected to anything float and will act like you saw.
The fact that most pins work correctly as outputs suggests that the board is fine to me.
Right, which is why I was using the internal pull-ups. I'm clearly not doing that wrong since some pins work as expected when I do that.