You need to distinguish between two types of conversion. Integral promotion is performed on a single operand. In the case of a shift operation, both operands are subject to integral promotion, independently of each other. The type of the right operand does not affect the type of the promoted left operand.
For binary arithmetic (not shift) operators, the "usual arithmetic conversions" are performed. In this case, the type of one operand can affect the type of the converted left operand. In particular, if one operand is unsigned int or a longer unsigned type, and the other operand has a signed type, then the other operand will be converted to unsigned. However, an expression of the form byte * byte will still be treated as int * int with the "unsigned" attributes discarded. See section 5.0.9 of the 2003 standard.