the atmega168 datasheet says with a 16mhz xtal, there is -3.5% error for 115200 baud rate. I need to interface with a AVRcam, which only uses 115200 baud and uses a atmega8 with a 17.7mhz clock. Will it work?
Since you've already found the serial clock info in the data sheet, check and see both what the error is and which direction the error is on a 168 at 16MHz and a mega8 at 17.7MHz. If they both tend to be a few % high, maybe they'll match.
I would be tempted to just try it, and if it doesn't work I would dig into the information on changing arduino clock rates, get an extra 17.7MHz crystal, and run the arduino at 17.7MHz.
-j