DMM = digital multimeter. Analog ones do just fine, too.
Again the output you claim to see and the sketch you post do not match up at all. I don't see how that sketch, which prints lines of 13 digits all being 1s and 0s, produces an output of just four digits that are not 1s and 0s.
Reading the spec sheet, several more issues come up.
For the inputs 1, 2 and 3 of the encoder: a HIGH is 10-30V. You provide 5V, very likely that is NOT read as a high signal by the encoder. You will have to add some circuits to get to a high signal here. The easiest is an NPN transistor with pull-up resistor that can pull the signal to the supply voltage of your encoder (12V). You will reverse logic this way: Arduino pin LOW means a HIGH signal to the encoder, and the other way around.
Signal durations on those pins must be >10 ms and reaction time of the encoder is up to 1 ms. Do keep those times in mind when programming.
The interface (the outputs of the encoder) are push-pull and short circuit protected, which is great. This means most likely your way too small resistor values did not destroy the encoder, but rather caused it to shut down or simply not do anything. They're also push-pull which means all you really need is a diode to interface with them.
The input: set your Arduino pin to INPUT_PULLUP. The diode stops any voltage >5V reaching your pin, while allowing the encoder to pull it low. The internal pull-up resistor makes sure your pin is pulled HIGH when the encoder's output is HIGH./ Much simpler and safer than messing around with voltage dividers.
The output: when your Arduino pin is HIGH, current flows through R26 switching on Q5 and the input of the encoder is pulled LOW. When your Arduino pin is LOW, the transistor blocks, and R19 pulls up the pin voltage to +12V (or whatever voltage you use to power the encoder - that's the voltage you want here). So you just have to remember to reverse the logic in your code.
Please note: none of the values of the resistors are critical, but you have to stay in this order of magnitude. The diodes are simple small signal diodes, both the common 1N4148 or 1N400x will do great. The transistor is a basic NPN small signal transistor, just about any NPN will do here.
And of course grounds need to be connected.