Probably what's happening here is that you're specifying baud rates high enough that you can't get UART baud rate close enough (it's derived by dividing the system clock, so as you get close to the minimum, it's harder to get close to the baud rate you need - google Wormfood AVR baud rate - it's got a chart that calculates how close you can get to various baud rates based on the system clock.
However, when you are using a board with the 16u2 as the USB adapter, both the USB serial adapter AND the 328p are trying to get to the same baud rate, subject to the same limitation (system clock is 16mhz on the 16u2 as well). So the '328p's baud rate will be wrong on both boards - but with 16u2 as serial adapter, the serial adapter's baud rate will also be wrong in the same way.