I think the issue is that % returns different values for negative inputs to an and? Certainly at a quick glance the code in SoftwareSerial::available () worries me - intermediate results are not cast to unsigned so will potentially be negative?
Anyway there's no need for % at all, each use can be replaced by a conditional subtraction of _SS_MAX_RX_BUFF from the variable that should be wrapping-round.