Well, with the original code (using the toggle) I find that OCR=1 generates 1/2 of 16384Hz (interestingly, so does OCR=0). At least, it should. I'm reading 8190, which is ~250ppm off—way more than the ~40 expected from the temperature. That could well be errors in the DMM—I don't really see what else, unless the crystal is picking up parasitic capacitance through the air from the breadboard rails underneath it.
That translates to 21 seconds per day slow, if I define a second as 8292 interrupts. (Or I could just code it as 8190, and assume that my meter isn't off.) That's not ideal, but it's not the second-every-five-minutes loss I was having before.
One thing I've determined: The calculator I linked to earlier rounds the answer it spits out—when I say I want 4 kHz, for example, it tells me I can use no prescaler and OCR=8 and have 0% error, but the fact is I will have error because everything's in factors of two. Nominal 8 kHz is actually 8192 Hz. This is possibly why the clock was so slow earlier.