I put the scope across the encoder outputs and go a lot of noise with the impressions of some square(ish) waves when rotating. More importantly, the peak voltage was less than a volt, so that made me realise the circuit is wrong because I didn't understand the term open-collector.
I'll report back soon, really busy with work at the moment.
Full schematic and code will be posted when it finally works 