program memory math strange behavior

AVR-GCC is essentially a compiler for a 16 bit processor; int is 16 bits. That means all expressions less than 16 bits are automatically promoted to 16 bits. For expressions that are expected to have a 32 bit result and start with 16 bit operands you, the developer, are responsible for the promotion.