but before the opto-couplers u could test what happens when u turn on the motor manually... with and without separated power supplies...
1. If you haven't already done so, connect a large decoupling capacitor (say 1000uF) between the motor supply +ve and ground, close to the motor H-bridge. Use the negative side of the capacitor as a star point to connect the ground lines to the H-bridge (which should be as short as possible), the 24V power supply, and the Arduino. The idea is to prevent the motor current inducing any voltage on the ground lines.
2. Keep the motor wires well away from the wires to the IR sensor.
3. If you have a spare ground pin on the Arduino, connect the ground side of the sensor to that and not to any other ground.
4. Use shielded cable between the IR receiver and the Arduino, shield connected to ground pin in point 3 above.
5. An external pullup resistor between pin 19 and +5v will reduce the sensitivity of that pin to noise. Try 2k2 or 4K7. Too low a value will stop the sensor working.
6. A capacitor between pin 19 and ground will also reduce the sensitivity to noise. Try 0.1uF. Too high a value will reduce the speed at which the sensor responds.
Why have you got a 10K in series with the step and direction pins to the controller? That is making them more prone to picking up interference, why can't you just connect the pins directly?
Connect the 1000uF capacitor before the H-bridge, that is, across the power (for the motor) and ground pins. 2k2 = 2.2 kohms (2200 ohms), 4K7 = 4.7 kohms. I think you mentioned that you already had such a capacitor connected about 4 inches from the H-bridge, however it should be closer than that.
When you connected the capacitor between pin 19 and ground, which ground did you connect the capacitor to? It should be grounded very close to the Arduino, to ensure that it doesn't pick up any noise induced in ground wires due to the motor current.
Are you certain that the power supply is capable of supplying the current needed by the motor at startup? If it isn't and the same power supply is powering the Arduino, then the Arduino could be resetting. Would that account for what you observe?
I was just thinking about this. My sensors are configured to ground the pin normally and to open the circuit when triggered by a flag (pin19 is using a RISING edge). In this scenario a pullup resistor would make the sensor read "on" all the time would it not?
1. Transient on the power that supplies the IR emitter. You said you had tried isolating the power to the emitter - what do you mean by this? Do you mean you tried driving it from a separate battery or wall wart?
4. A transient in the ground wiring or power supply to the Arduino causing the Arduino to malfunction. However, you said that it can't be resetting. Have you checked that nothing observable happens if you press the reset button? Even if your code does nothing, output pins become floating after a reset until the pinMode call is made, also internal pullups get disabled until they are enabled again. Either of these could cause something to happen.
I tried disconnecting the ground between the 24v power supply and the rest of the circuit, as Riddick suggested, and the problem did NOT occur when starting the motor manually.
maybe the cables between the arduino and the sensor act as an antenna?what if u use really short cables?or what if u put those cables far away from the motor?
The circuit you posted shows the emitter being driven from 3.3v through a 470 ohm resistor. Even a 1v transient might be enough to cause a glitch. I suggest driving the emitter from 5v instead. Also the resistor looks a little high to me, I thought these devices usually wanted around 10mA.