For an optical encoder you would have four pins: +5, Ground, A and B.
Is this a mechanical encoder with only three terminals? In that case you could connect the common to Ground and use pull-up resistors (either the built-in ones or external fro the input pin to +5). Alternatively you can connect the common to +5 and use pull-dowm resistors from the input pins to Ground. Without pull-up or pull-down resistors the switches are switching between +5 and floating or Ground and floating. When left floating the input could read HIGH or LOW randomly.