Ok I removed the LEDs (and corresponding resistors) and got the same Serial Monitor output. I decided to unhook everything, except for the IR Led, which i know works, and the strange thing is that the Arduino is still giving the same Serial Monitor output, even with nothing connected?!
Heres a code example i tried, followed by the output:
/* Read Quadrature Encoder
* Connect Encoder to Pins encoder0PinA, encoder0PinB, and +5V.
*
* Sketch by max wolf / www.meso.net
* v. 0.1 - very basic functions - mw 20061220
*
*/
int val;
int encoder0PinA = 2;
int encoder0PinB = 3;
int encoder0Pos = 0;
int encoder0PinALast = LOW;
int n = LOW;
void setup() {
pinMode (encoder0PinA,INPUT);
pinMode (encoder0PinB,INPUT);
Serial.begin (57600);
}
void loop() {
n = digitalRead(encoder0PinA);
if ((encoder0PinALast == LOW) && (n == HIGH)) {
if (digitalRead(encoder0PinB) == LOW) {
encoder0Pos--;
} else {
encoder0Pos++;
}
Serial.print (encoder0Pos);
Serial.print ("/");
}
encoder0PinALast = n;
}
and the output which steadily streams

Code example 2
/* read a rotary encoder with interrupts
Encoder hooked up with common to GROUND,
encoder0PinA to pin 2, encoder0PinB to pin 4 (or pin 3 see below)
it doesn't matter which encoder pin you use for A or B
uses Arduino pullups on A & B channel outputs
turning on the pullups saves having to hook up resistors
to the A & B channel outputs
*/
#define encoder0PinA 2
#define encoder0PinB 3
volatile unsigned int encoder0Pos = 0;
void setup() {
pinMode(encoder0PinA, INPUT);
digitalWrite(encoder0PinA, HIGH); // turn on pullup resistor
pinMode(encoder0PinB, INPUT);
digitalWrite(encoder0PinB, HIGH); // turn on pullup resistor
attachInterrupt(0, doEncoder, CHANGE); // encoder pin on interrupt 0 - pin 2
Serial.begin (57600);
Serial.println("start"); // a personal quirk
}
void loop(){
// do some stuff here - the joy of interrupts is that they take care of themselves
}
void doEncoder() {
/* If pinA and pinB are both high or both low, it is spinning
* forward. If they're different, it's going backward.
*
* For more information on speeding up this process, see
* [Reference/PortManipulation], specifically the PIND register.
*/
if (digitalRead(encoder0PinA) == digitalRead(encoder0PinB)) {
encoder0Pos++;
} else {
encoder0Pos--;
}
Serial.println (encoder0Pos, DEC);
}
/* See this expanded function to get a better understanding of the
* meanings of the four possible (pinA, pinB) value pairs:
*/
void doEncoder_Expanded(){
if (digitalRead(encoder0PinA) == HIGH) { // found a low-to-high on channel A
if (digitalRead(encoder0PinB) == LOW) { // check channel B to see which way
// encoder is turning
encoder0Pos = encoder0Pos - 1; // CCW
}
else {
encoder0Pos = encoder0Pos + 1; // CW
}
}
else // found a high-to-low on channel A
{
if (digitalRead(encoder0PinB) == LOW) { // check channel B to see which way
// encoder is turning
encoder0Pos = encoder0Pos + 1; // CW
}
else {
encoder0Pos = encoder0Pos - 1; // CCW
}
}
Serial.println (encoder0Pos, DEC); // debug - remember to comment out
// before final program run
// you don't want serial slowing down your program if not needed
}
/* to read the other two transitions - just use another attachInterrupt()
in the setup and duplicate the doEncoder function into say,
doEncoderA and doEncoderB.
You also need to move the other encoder wire over to pin 3 (interrupt 1).
*/
The output just prints startCR on the screen and then nothing happens afterwards
Code example 3
#define encoder0PinA 2
#define encoder0PinB 3
volatile unsigned int encoder0Pos = 0;
void setup() {
pinMode(encoder0PinA, INPUT);
pinMode(encoder0PinB, INPUT);
// encoder pin on interrupt 0 (pin 2)
attachInterrupt(0, doEncoderA, CHANGE);
// encoder pin on interrupt 1 (pin 3)
attachInterrupt(1, doEncoderB, CHANGE);
Serial.begin (57600);
}
void loop(){ //Do stuff here
}
void doEncoderA(){
// look for a low-to-high on channel A
if (digitalRead(encoder0PinA) == HIGH) {
// check channel B to see which way encoder is turning
if (digitalRead(encoder0PinB) == LOW) {
encoder0Pos = encoder0Pos + 1; // CW
}
else {
encoder0Pos = encoder0Pos - 1; // CCW
}
}
else // must be a high-to-low edge on channel A
{
// check channel B to see which way encoder is turning
if (digitalRead(encoder0PinB) == HIGH) {
encoder0Pos = encoder0Pos + 1; // CW
}
else {
encoder0Pos = encoder0Pos - 1; // CCW
}
}
Serial.println (encoder0Pos, DEC);
// use for debugging - remember to comment out
}
void doEncoderB(){
// look for a low-to-high on channel B
if (digitalRead(encoder0PinB) == HIGH) {
// check channel A to see which way encoder is turning
if (digitalRead(encoder0PinA) == HIGH) {
encoder0Pos = encoder0Pos + 1; // CW
}
else {
encoder0Pos = encoder0Pos - 1; // CCW
}
}
// Look for a high-to-low on channel B
else {
// check channel B to see which way encoder is turning
if (digitalRead(encoder0PinA) == LOW) {
encoder0Pos = encoder0Pos + 1; // CW
}
else {
encoder0Pos = encoder0Pos - 1; // CCW
}
}
}
and the output prints really fast

Here's a photo of the setup, i've disconnected the arduino and its really strange why its giving the same output as when I connected the phototransistor.

I can't find a datasheet for this transistor nor know its model number. The ones i took out from other mice have EL embossed on the top, which probably stands for Everlight electronics ltd. This one has LT embossed on the top.
Looking at the leads , i'm now noticing that only one of them is thicker, perhaps this phototransistor is not in the usual pin configuration

http://www.seattlerobotics.org/encoder/200311/johnson/Mousebot.html
Theodore Johnson explains "The three-pin detector consists of two photodiodes arranged vertically. The outputs must be pulled to ground by, e.g. a 10K Ohm resistor." And provides the following diagram:

Steve Kamerman uses a 5 pin unit, where the first 3 pins are for the phototransistor, and the last 2 are to power the IR led
Pin 1: Photo Sensor 1
Pin 2: Vcc
Pin 3: Photo Sensor 2
Pin 4: IR LED +
Pin 5: IR LED -
He comments "There are two 10k resistors. First, connect pin 1 directly to arduino input 2, then put the 10k from ground to input 2. This creates a voltage divider since pin 1 will be slightly positive and the resister will pull it slightly towards ground. Repeat for the other encoder output – pin 3 to input 3, 10k from ground to input 3."
http://www.mcmanis.com/chuck/robotics/projects/lab-x3/quadratrak.html
Here's an image of a Rotary Encoder setup from Chuck McManis' site

Here's another setup from Hobbytronics

Here's a pinout of the Everlight Electronics dual phototransistor from the datasheet

I'm also going to stare at the above information long enough, now that its all in one place, and see if the issue will sort itself out that way. But further help is much appreciated too. Thanks!