You don't know the pins output? Perhaps there is none.
Can you remove this PCB mouted at motor and send PCB underside photo?
Where did the motor come from? Can you post a link?
My guess would be:
- OUT1 from h-bridge
- OUT2 from h-bridge
- Logic power + from Arduino 5V
- Logic power - from Arduino ground
- Encoder output A to Arduino input pin
- Encoder output B to Arduino input pin
Maybe start by identifying the OUT1 and OUT2 wires. Can you access the motor terminals? I guess they might be soldered to the underside of the PCB. Use a multimeter on resistance mode to identify which wires are connected to those motor terminals. When you find zero resistance, you have identified the wire connected to a motor terminal. If you measure a resistance of a few tens to a few hundred ohms, you are measuring the motor winding resistance: try another wire. For any other wire, the resistance will be "infinite".
If you spin the motor, you might see a dc signal on the two motor wires.
Do you have a DMM?
Your mileage may vary but here is my limited experience:
I had a similar quest a while back. It wasn't a motor, but was a small circuit board with and opto isolator used as a counter via a slotted disc attached to a motor. The circuit board also had a limit switch and something else, though i can't remember for sure now.
Anyway, i was able to pull the board and see both sides of the circuit board so i could see the traces. Armed with very little knowledge and a multimeter i was able to figure out what was what. Starting with the LED (which only works with electric flowing one way) on the opto isolator and tracing the traces i was able to identify all grounds/commons and power to the board. Using that info, i was able to start figuring out which wires went where and finally got it figured out.
It was a learning experience for this new guy. It had 5 wires on it (i think) and i was able to figure out each one.
Is there a component on there that you know what is? Perhaps you can start with that.
Hi, @6jamal_mesmar77
Welcome to the forum.
Can you please post some images of the other side of the motor and the PCB.
Does the the PCB have any components on it?
Tom.... 
