@el_supremo: I didn't consider the IC socket, I soldered it in but maybe it's a bad joint? I do not have a capacitor across the 5V pin but I'm making a new board that does have it.
@jluciani: I am running everything off of a 6V sealed lead acid battery that is running through a low dropout regulator. I do not think that there is battery derating because the rest of the components are ok. This is the battery I'm using: http://www.batterysharks.com/MK-Battery-ES4-6-p/es4-6_b6-4.5.htm. As for the crystal, I'm using this one: http://www.mouser.com/Search/ProductDetail.aspx?R=AB38T-32.768KHZvirtualkey52750000virtualkey815-AB38T-32.768KHZ. It's 32.768 kHz and 12.5 pF, just as specified. I do not however, have a ground fill under the chip as seen in this picture:
Would that be a possible source of the problem? I thought the ground plane/pour was just there to reduce noise on any other lines.