Watch out. This loop has the potential for the 40-byte buffer to be written out of bounds.
Note that most terminal programs will append a CR (ASCII 13) and/or LF (ASCII 10) to the sent character array. I suspect this has something to do with your problem.
In tracing problems like these, start by printing the contents of buffer[] after receipt. You're now only printing the atoi-converted value, but have a look at what's really there. I suspect the CR+LF results in a second value of 0 being recorded for every Serial receipt.