Blink not in sync with two Arduinos. Is it supposed to be?

In my attempt to debug a problem, I went back and loaded the simple blink example to two Nanos. I powered both of them at the same time (from the same power source). For the first 10-20 seconds, they seem to blink in sync. But after that a time shift is noticeable. At time passes into the minutes, they are totally out of sync.
Did the same with 4 Nanos and the phenomena is the same.

Is this normal? What can explain this? Why they are not behaving exactly the same?

Just to stress it again - they start together, totally in sync. After a while they go out of sync.

It is normal for two boards not to stay in sync, due to tolerances in the components. Specifically, the system clocks do not run at exactly the same speed, so the two will drift apart over time.

What is the timing facility on that board? On the pictures on the main site it looks like a crystal, but some people say they have a ceramic resonator. I myself have never had a nano, so I don't know.

If it is a ceramic resonator, then yes that would be normal - the timing of a resonator can be wildly off compared to a crystal.

What if you do this with a blink without delay sketch ?

Each arduino board has its own crystal. The frequency of each crystal is not strictly the same. The clock signal frequency remains constant but phase slip.

To have a true synchronization you must use only one card with a cristal.
The second board receive the clock frequency of the first board.

Consult the microcontroller datasheet, the procedure is described.

majenko:
What is the timing facility on that board? On the pictures on the main site it looks like a crystal, but some people say they have a ceramic resonator. I myself have never had a nano, so I don't know.

If it is a ceramic resonator, then yes that would be normal - the timing of a resonator can be wildly off compared to a crystal.

I was curious about that too. The picture on the Arduino site shows a Nano v2.2, and appears to have a crystal. I wouldn't expect two crystals to lose sync in just 10-20 sec. But looking at the Nano v3.0 on Gravitech's site, the crystal is gone, apparently replaced with a resonator. If the OP has v3.0, then that would explain the observations.

Note to OP: Crystals are usually accurate to about 20 ppm, resonators to about 0.5% (5000 ppm), so that is a significant difference. I have three custom designed boards that use crystals that happen to blink an LED for 30 sec after a reset, and they appear to stay well in sync for that period of time.

Thanks all. The Nano is using a Murata resonator. I wasn't aware they are so different apart from crystals.
So bottom line is, it's normal, right?

szangvil:
Thanks all. The Nano is using a Murata resonator. I wasn't aware they are so different apart from crystals.
So bottom line is, it's normal, right?

Yep, it's normal.

That's why for anything with critical timing, like USB, you use a crystal not a ceramic resonator.

majenko, what do you mean? The Nano has a USB FT232RL chip and there is no problem using it...

szangvil:
I powered both of them at the same time (from the same power source)

I would call this an invalid assumption. While you are probably starting them as close as you possibly can, they are not starting at the same time.

It won't take long for the difference in start time to accumulate to the boards getting out of sync. The fact that there is some slop in the delay counter adds error too.

szangvil:
majenko, what do you mean? The Nano has a USB FT232RL chip and there is no problem using it...

According to the schematics, the FT232RL chip doesn't have any connection to any clock source at all. According to the FTDI website (though it's hard to be certain with their wording) the chip has its own internal clock generator.

Edit: The data sheet is more explicit:

Fully integrated clock generation with no external crystal required

szangvil:
In my attempt to debug a problem, …

What problem? Is it essential for the clocks to run at exactly the same rate? Why?

You have some options, one of which is that the processor can be configured to output its clock on one pin. And the other ones could read that clock in as their clock. That way they could all run at the same clock rate.

Alternatively, output a “sync” signal on a master processor, and have the others wait for it to arrive before continuing (eg. in loop).

Normally it shouldn’t matter, most protocols (eg. I2C, SPI, async) have provision built in for managing slightly different clock rates. I2C and SPI are self-clocked, and async serial can tolerate some drift.

Nick, the devices have a GPS module in them. They are both powered on at the same time and at the same location but they never lock on to the GPS signal at the same time... But I guess the problem (or behavior) is the GPS module itself since it, like the FTDI chip, has external clock.
I was wondering if it can be related to the serial communication between the atmega328 chip and the GPS module.

szangvil:
Nick, the devices have a GPS module in them. They are both powered on at the same time and at the same location but they never lock on to the GPS signal at the same time... But I guess the problem (or behavior) is the GPS module itself since it, like the FTDI chip, has external clock.
I was wondering if it can be related to the serial communication between the atmega328 chip and the GPS module.

Probably more likely related to the modules not finding the exact same satellites at the same exact time. The GPS signal is incredibly weak. There are lots of satellites to try for on lots of frequencies (channels). It's quite a complex operation getting and maintaining a GPS lock.

What has this got to do with the blink example not staying in sync?

What do you really want to achieve here?

That seems to not be unusual. One antenna may be a micro meter shorter then the others, so one may receive signals slightly better. What is the times you are looking at (10 - 30 seconds)? Different times of the day (or really any time, the satellites will be in different locations), cloudy conditions are different.
I have no clue of the algorithm used to check for all the different satellites, or If both units go through the same sequence searching.

Does one unit always beat the other one? By how much?

majenko:
The GPS signal is incredibly weak

Indeed it is, a normal figure for received signal strength is around -130dBm, that's -130dB referenced to 1mW. Converting that into Watts, it is an approximate power of 1x10-16 or 0.1 femto watts.
On top of that, GPS uses a frequency of around 1.5GHz, or a wavelength of 20cm. So if your GPS recievers are even a small distance apart from each other, then the signals received could potentially be out of phase of each other, so basically the two GPS recievers will be receiving different signals at any given moment in time (well, the same signal, but with a phase difference). Its no wonder they don't get a lock at the same time.