How's it wired?
Are you powering it through a regulator? Regulators have a "drop out" - you lose at least a volt across most regulators, so if you're putting 3.4v into a 3.3v regulator, you're probably only getting 2.x volts out, probably bringing the chip out of spec.
Do you have the decoupling caps in place (2 0.1uf caps, one between Vcc and Gnd, other between AVcc and Gnd, ceramic caps, located right next to the chip)? Some guides omit them, despite the fact that the '328p doesn't function reliably without them - and lowering the operating voltage would magnify the effect of this.