I like your approach that keeps this as non-blocking and free running as possible. Haven't looked closely at your code yet, but I suspect there could be some improvement(s) to be had in the circuit.
I guess you're using a 5V MCU to drive the PWM dimmer circuit (because of the total VF of both LEDs)
What color is the visible LED?
What I would like to do is enhance it to do things like Auto detect AC frequency and
fix the glitch when it gets close to the 100% point (1024).
With regards to the glitch, I wonder if this is a hardware or software issue.
Do you have a scope trace or any more information?
What type of AC load are you using?
Perhaps the AC load needs a snubber circuit, as not having one on an inductive (or capacitive) load might me the cause, or contribute to the glitch.
Maybe a resistive load (for testing) would make a difference?
EDIT: Software
Re: datasheet 14.7.3 Fast PWM Mode
The extreme values for the OCR0A register represents special cases when generating a PWM waveform output in the fast PWM mode. If the OCR0A is set equal to BOTTOM, the output will be a narrow spike for each MAX+1 timer clock cycle. Setting the OCR0A equal to MAX will result in a constantly high or low output (depending on the polarity of the output set by the COM0A1:0 bits.
OCR1A = OCR1A = ICR1 - dutyCycle; // invert 0-1024 range
If you want 10-bit range, wouldn't the max setting be 1023? (0-1023 = 1024 values)
EDIT: back to Hardware
There's room for improved accuracy of the zero-cross circuit. The H11L1 looks interesting.
Another EDIT:
...I noticed that the incandescent light I was using to visualize my output flickered when events occurred...
There's significant EMI when switching an incandescent load (filtering is needed). Really, a lot depends on the type of load being switched. If this is intended for use as an LED driver, the circuit would need to be more complex.