The brightness of the LED, amount of light blocked by the BB and ambient light will affect the sensor. Could you have reflections from the mounting bracket allowing light to bypass the BB?. Make sure the LED and sensor are aligned with each other and the BB's path. Is there be any play or vibration in the mount? Anything you can do to reduce external light into the sensor will probably help.
Increase and decrease the brightness of the LED until the sensor reads 'dark' then increase the brightness a little. That way it will only take a small change in light to trigger the sensor.
I assume the sensor is digital. If it is analog and you are using a comparator circuit You can adjust both the brightness and the threshold so the sensor is in the middle of it's range which is probably the most sensitive and linear region.