Ya, that was the problem. It works fine now after changing one to read the left and the other the right sensor, I must have copied/pasted that part.
As for the floats, I'm multiplying the summed sensor values by a decimal so I tried using floats for those variables when I was troubleshooting, but it's probably not necessary. A rounded integer would probably work about the same.
The averaging didn't help performance that much really, I think it's not so much a problem of noise but low light sensitivity. The photodiodes I'm using are supposed to produce a current down to ~10 lux. It works fine in bright light or shining a flashlight in front of it but not dim room lights. I think I'll try better photodiodes, something like this:http://www.thorlabs.com/thorProduct.cfm?partNumber=FDS100
Or, just for kicks I thought it might be cool to try and run photomultiplier tubes so it could drive around it VERY low light (e.g. moonlight coming in the window), of course you'd have to be careful not to cook them in high light conditions so probably not really worth the trouble.
This stuff is fun.