Multiple arduino staying in sync

I’m planing some outdoor lighting LEDS controlled via Atmega328P, with 16mhz crystals. I will have several in one location that need to be in sync. I can’t have 1 chip controlling all of them due to logistics. Each Atmega328P will control about 6, but I need them to be in time.
They will all be powered by the same 110vac power cord, each one using this AC/DC converter.
https://www.digikey.com/product-detail/en/recom-power/ROF-78E5.0-0.5SMD-R/945-1690-1-ND/3593416

They’ll run for about 4 hours when they come on in the evening. I’m thinking they should all stay within 10-20 Millis of each other even by the end or 4 hours? Enough that a person wouldn’t notice the difference?

If you could get them to all start at exactly the same time and have all exactly the same clock frequency, then yes. But to be within 20msec after four hours, then all 230,400,000,000 (16Mhz * 60 * 60 * 4) clock pulses must be within 0.86 femtoseconds (0.86x10^-15) of each other. I'm not thinking this is going to work well. My suggestion is a sync pulse sent periodically from one board to all the others. Additionally, the other "slave" boards hold until the first pulse is received.

Thanks. I didn't realize they would get off that much. having them sync via a connecting wire is not an option, but I did find that the normal crystal I use is 50ppm, but I found this SMD tiny one that is 10ppm, which, if my calculations are correct, will put it in the range of +/-150 millis, after 4 hours, which won't be too noticeable to the eye, especially since the Outdoor lights will fade from one color to the next. Temp change shouldn't matter, since they'll all be the same temp.
thanks.

It should not be difficult to test on the bench. Get two Arduinos and run the Blink sketch for a few hours. Maybe wrap one in bubblewrap or other insulation to simulate the worst-case temperature difference.

If you're color-fading outdoor lights and not trying to make them flash in unison then several seconds error could be tolerable. It will only be a problem at Christmas when you do want flashing lights.

If the different processors are going to execute different lighting scripts, although in sync, the crystal lock ain’t going to work, because each of the scripts will execute slightly differently, based on where they are within each script.
if()s, switch()s and other calls will be timed differently in each node at slightly different moments in time - growing over time.

You really need some form of heartbeat to keep them in sync.

lastchancename:
If the different processors are going to execute different lighting scripts, although in sync, the crystal lock ain’t going to work, because each of the scripts will execute slightly differently, based on where they are within each script.
if()s, switch()s and other calls will be timed differently in each node at slightly different moments in time - growing over time.

You really need some form of heartbeat to keep them in sync.

But if the transitions are timed from the millis() function, I bet it will stay close enough in sync.

They’ll all have the same script.
I’ll post my results when me boards come in.

Of course, since they are mains powered from the same ‘city’, you could derive a heartbeat with a 50/60Hz zero-crossing detector on the AC... the ‘global* heartbeat will drift by the same (tiny) amount over time but stay ‘in sync’

They still run at 16 MHz, but each ‘frame’ is resynchronised every 15-20mS.

lastchancename:
Of course, since they are mains powered from the same ‘city’, you could derive a heartbeat with a 50/60Hz zero-crossing detector on the AC... the ‘global* heartbeat will drift by the same (tiny) amount over time but stay ‘in sync’

They still run at 16 MHz, but each ‘frame’ is resynchronised every 15-20mS.

I had wondered about such a thing.

They could still get out of sync right at start-up. Your expecting the 6 arduinos to start the first instruction within 0.1 seconds of each other.

True, the bootloader timeout and other variables could be minimised, but never completely.
An interesting challenge!

The zero crossing would be easy enough with an opto across the AC secondary.

if the controllers are in a single location, how about at power on, they go into a ‘waiting’ state when first initialised, then started in sync by a simple photo flash or other short pulse event.

Zero crossing still only gives you 0.5s intervals. I don't know the variability of the startup sequence to know if the will be ready to see the same zero crossing. It could put some 0.5s off. I'm still thinking a "master" is needed to start the sequence. If you go that far, you might as well re-sync periodically.

I may have missed the point, but zero-crossing in 50Hz would provide 100 Hz ticks (using both leading and trailing edges)

That’s 10mS, or 20mS if only sensing one edge.
But I agree the startup sync may still be an issue between multiple non-connected boards. Hence the added ‘start’ pulse...
.

Why? If the Arduinos all have the same bootloader then what could possibly cause variation in the time to reach setup()?

MorganS:
Why? If the Arduinos all have the same bootloader then what could possibly cause variation in the time to reach setup()?

The tolerance of the capacitors on the reset lines.

You could add a GPS module to each arduino to sync times. Without a wired connection this might be the easiest option, although not cheap.

If you’re going wireless, there are many options available.

I’d probably go with an ultra cheap low-latency 433 link (one Tx, many Rx) so one acts as a heartbeat master and all the others follow it.

If you have a data link, that also offers other capabilities which may not be needed now - but maybe something in the future...

BUT.
I still like the AC zero-crossing ticks, with an externally generated pulse to resync...

STOP PRESS...
I just realised the resync action could be a missing cycle on the AC to tell each node it’s time to ‘go’.
That would be a lot easier if you distribute the low-voltage AC to the nodes, rather than the full mains 230V

How about using the mains to distribute the synchronising signals. Modulate these at say 100KHz and couple into the mains with two small capacitors.