Mysterious random i2c fault across different devices

Hello,
the last couple days I have been tracking down a sporadic bug where communication with an i2c peripheral suddenly stops. I narrowed it down to analogRead() calls for A0 followed by A1 I do every loop() iteration. With these reads present, it can happen that at a certain point in an i2c transaction when an ACK or NACK is expected, the data line is being kept low (top: normal, bottom: abnormal end):

Note that in this case, endTransmission() returns 2 but this isn't strictly true, because the logic analyzer output shows that neither ACK nor NACK was received (2 referring to the latter). It's probably another internal error presented as code 2.

The bug can happen a minute after startup or within an hour after hundreds of thousands of otherwise successful transactions. Althought the bug is probabilistic, the situation where it occurs is always precisely at the ACK/NACK bit in a write transaction. It seems to only affect Wire and not Wire1.

Any ideas?

Arduino UNO R4 Boards 1.6.0
Arduini IDE 2.3.10

Thank you

I have no idea what hardware you are using, so it is difficult to give a specific answer. On AVR processors I routinely use analogRead() during setup and have never had a problem.

However, I never use analogRead() on the I²C pins after the I²C interface has been initialized. The I²C peripheral takes control of those pins, and mixing ADC operation with active I²C communication can create unexpected behavior.

Also be aware that some devices and peripherals can latch up if pins are driven when the device is unpowered or not configured as expected. Make sure the power sequencing, pin states, and pull-ups are correct before assuming it is a software problem. A clean power supply is also critical on many of the I2C parts.

Post the exact processor, board, wiring, and an annotated schematic if you need a specific answer.

This was posted in the Uno R4 WiFi category so that should answer (part of?) the question :wink:

if you only test the I2C device (without calling analogRead() or accessing any other devices) does it work OK?
what I2C device and are you using a library or have you implemented the I2C code yourself?
do any other I2C devices have the same problem?
as @gilshultz requested post a schematic and the code used for testing

Who is holding the line low, the master or the slave?
If you disable the I2C after the fault occurs does the line return high?

Thanks for the responses so far - luckily I can edit the title because it turns out I was fooled by randomness and analogRead() has nothing to do with the problem, it perhaps just made it more likely to occur due to timing changes.
I managed to reproduce it with a bare minimum sketch:

#include <Wire.h>
#include <RTClib.h>

void setup() {
  Serial.begin(9600);
  while (!Serial);
  Wire.begin();
}

void loop() {
  Wire.beginTransmission(0x68);
  uint8_t zero = 0;
  if (Wire.write(&zero, 1) != 1) {
    Serial.println("write() failed!");
    while(true);
  }
  uint8_t result = Wire.endTransmission();
  if (result != 0) {
    Serial.print("endTransmission() failed with code ");
    Serial.println(result, DEC);
    while(true);
  }
}

Post the exact processor, board, wiring, and an annotated schematic if you need a specific answer.

Components involved:

I am aware that those are cheap modules possibly prone to defects but what are the odds that both my DS3231 and DS1307 module show exactly the same behaviour?
I have also swapped the Arduino for a brand new one and observe the same behaviour. It doesn't matter if I hook up the modules via Qwiic connector or SDA/SCL, same issue. I usually have only one attached, doesn't matter if on Wire or Wire1, same issue.
One thing I notice is that it used to run for weeks without problems at first but now I can reproduce the problem within 5 minutes max, so it appears to be gradually getting worse.

I never use analogRead() on the I²C pins after the I²C interface has been initialized

I am using SDA/SCL for I2C and A0/A1 for analog input measurements, but that's moot now anyway.

A clean power supply is also critical on many of the I2C parts.

It happens when I provide power via USB cable from my PC and also when I provide power via a 12V Meanwell PSU.

what I2C device and are you using a library or have you implemented the I2C code yourself?

I use the official Wire class.

Who is holding the line low, the master or the slave? If you disable the I2C after the fault occurs does the line return high?

I have tested this by unplugging SDA from the peripheral during the error condition, connected it to VCC of the Arduino with a 3.3k Ohm resistor in between and did measure it HIGH, suggesting that it's the module that pulls it LOW. It was a bit of fiddling and I might have disturbed VCC/SCL so I don't know if that caused an I2C reset internally and if my measurement was actually probing the fault condition or its recovery.
But I will do this again with a better setup and also check on the module's end.
Looking at the waveform and the bit pattern, the only realistic scenario I see in case the module pulls it LOW is that it does so since the previous ACK, got stuck there and it's only apparent at the next ACK because the bits in between were all coincidentally zero.
There is one interesting measurement I took with a module on Wire1 and another I2C peripheral on Wire2:

Notice the gap (about 65ms) on Wire1 right when the fault on Wire occurs. No such gap was present earlier in the recording. First I thought that would speak against the peripheral being the culprit because why would it influence the other Wire1 like this. But then I thought the opposite: Arduino is likely busy-waiting until a timeout and unable to service Wire1 during that time.

can you give links to the actual DS3231 and DS1307 modules you have?

which device did you connect to SDA/SCL (uses 5V logic) and the Qwiic connector (uses 3.3Vlogic)

did you try any DS3231 and DS1307 libraries?

can you give links to the actual DS3231 and DS1307 modules you have?

The 1307 I ordered from an electronics store and the 3231 is from Amazon - I did not manage to find a proper original manufacturer datasheet associated with them. It's apparently "standard cheap clock modules" that "everybody is using". Pretty frustrating.

which device did you connect to SDA/SCL (uses 5V logic) and the Qwiic connector (uses 3.3Vlogic)

They both work on Qwiic, for 1307 the 3.3V are out of spec though, so I wouldn't use it permanently. The 3231 operates under both voltages.

did you try any DS3231 and DS1307 libraries?

Yes originally, the default(?) RTClib, at least until I noticed that it doesn't do any i2c error checking and happily reports an uninitialized buffer as datetime... Those libraries don't do anything out of the ordinary, really only requesting the datetime bytes and checking clock halt bits, which I now do directly - with proper checks.

Posting in the UNO R4 WiFi section does not tell us which board the OP actually has. Please identify the exact board you are using, preferably with a link to its technical information. Also run an I²C scanner and post the results. Without knowing the exact hardware and the addresses actually present on the bus, we are mostly guessing.

There are actually two main Arduino UNO R4 boards:

  1. UNO R4 Minima — the simpler version. It uses the Renesas RA4M1 32-bit ARM Cortex-M4 processor, runs at 48 MHz, and uses 5 V logic.

  2. UNO R4 WiFi — uses the same RA4M1 processor but adds an ESP32-S3 for Wi-Fi and Bluetooth. It also has a 12×8 LED matrix and a Qwiic I²C connector.

For most basic Arduino programming, the two are very similar, but they are not identical boards. The WiFi version has additional onboard hardware and different board-level connections.

There are also different production revisions, but from the standpoint of selecting a board in the Arduino IDE, the important choices are the UNO R4 Minima and UNO R4 WiFi.

Does the DS3231 have an internal alarm set that initiates an interrupt?

Yes. The DS3231 has two independent internal alarms, Alarm 1 and Alarm 2. Either alarm can be programmed to generate an interrupt when the RTC time matches the programmed alarm conditions.

The interrupt appears on the DS3231 INT/SQW pin. This is an open-drain output, so it requires a pull-up resistor. When an enabled alarm occurs, the DS3231 pulls this pin LOW.

Alarm 1 can be set with resolution down to one second. Alarm 2 has one-minute resolution. The alarms can also be configured for different matching conditions, such as every minute, a particular minute and hour, or a particular time and date.

One important detail is that the interrupt does not automatically clear itself. When the alarm occurs, the corresponding alarm flag in the DS3231 status register is set. Your program normally responds to the interrupt and then clears that flag. Until it is cleared, the INT/SQW output can remain LOW.

The alarm function is internal to the DS3231. The processor does not have to continuously compare the time itself, and the alarm will continue operating from the backup battery if the main power is removed.

FYI it also has a die temperature sensing capability and can be adjusted for errors in the master oscillator.

I am in the process of writing a library for this part, let me know if interested. It will be a few days until ready. I am in the process of testing the code and writing the manual for it.

how long does it take the code of post 6 to fail?

I have run the program for two hours on a Uno R4 WiFi with a DS3231 connected to SDA/SCL without any error report

what is the error message when your device fails?

how do you power the UNO R4?

EDIT: been running for 12 hours now

EDIT2: been running for over 24 hours - no problems

What is this glitch?
If ACK is LOW and next bit is zero, this should not be there.

This puzzled me a bit:

The syntax permits the following:

Wire.write(value)

Wire.write(string)

Wire.write(data, length)

You appear to be using the last option, but this requires the following:

data

: an array of data to send as bytes. Allowed data types:

    byte[]

length

: the number of bytes to transmit. Allowed data types:

    size_t

But this code passes the address of a single uint8_t type variable zero?

If passing a single byte (or uint8_t) then just use, omitting the ampersand and data length:

if (Wire.write(zero) != 1) {

When sending multiple bytes, then the last option becomes more suitable, e.g.:

  size_t dataSize = 10;
  byte data[dataSize] = {0};
  //
  // Some code to place data into the array
  //
  if (Wire.write(data, dataSize) != 1) {
     ...
  }

Perhaps passing the address of a scalar variable is causing something unexpected?

Note: arrays are automatically passed as references so the '&' is not required there either, although I can see from the compiler warning how you might have arrived at the thought to add it....

Think the Wire.write() will always return 1 as this call puts a (single) value in the internal buffer. The actual write is done with the Wire.endTransmission().

When the I2C stops, is there a message printed?

I have run your program of post 6 for over 30 hours on Uno R4 WiFi with a DS3231 connected to SDA/SCL without any error report

how do you power the setup?
what are the connections between the Uno R4 WiFi andDS3231? e.g. DuPont connecting wires which are a source of poor connections and intermittent problems? if so how long are the wires?

post a photo?

Thanks for pointing out the alarm capability - in theory there could have been some kind of disturance or glitch where an alarm being raised causing an i2c glitch but unfortunately it isn't it. First of all, none of the other pins are connected, no alarm is configured and the other simpler DS module with the same behaviour does have alarms.

I am pretty confident that this is a normal i2c waveform. The "glitch" is roughtly 200ns long and shows the peripheral releasing SDA after holding it down during the 9th pulse to indicate ACK. Since the master wants to transmit a zero bit thereafter, it immediately pulls it down again.

No, that's actually fine. write() just takes a const uint8_t * and I'm passing the address of a local variable of type uint8_t. All the function sees is an address and a size, it doesn't know and doesn't need to know if that address refers to a single value on the stack or an array somewhere. As long as it's honouring the size parameter and does not attempt to read anything beyond the address provided, everything is good.

Thanks, yes, it can be that the write() check is superfluous. The error that is printed is consistently endTransmission() failed with code 2.