What is the actual voltage you measure on the pin?
The only values you can possibly get from an analogRead() is 0-1023. If you see negative results, there's something wrong with the maths.
The first suspect is the variables (ints can go negative - if you use unsigned int and you still see negative values as you suggest there's something impossible going on).
The second suspect is the algorithm itself. There's probably something wrong there. Start by printing out intermediates.