I2C on slave device hangs when data is received on slaves UART RX.
I'm developing a device that has a master Arduino Mega2560 and two slave Arduino Mega2560s. The slaves each communicate with their own MPU6050 IMU sensor over I2C (using a logic level converter) and are located remote (~5m) from the master. The master and slaves communicate over RS485, using a MAX485 UART to RS485 breakout board to convert UART TTL to RS485. The master will have two of these boards, so that each slave has it's own seperate connection to the master. The master will request motion data at 20Hz from the slaves, compare the values and then drive some LEDs based on these values. The request from the master is in the form of the character "1" being sent to the slave using Nick Gammon's RS485 library. I'm using the blocking version on the master, and the non-blocking version on the slave.
Libraries being used are as follows;
The schematic for the project, which shows only one slave, can be seen below. The cable used in my test setup is 3m CAT5E, unshielded twisted pair. I'm using just the one pair. My pullup resistor configuration results in a sink current of ~2.5 mA for the driving I2C device. Maximum is 3 mA so I can't go much lower with my pullup resistor values.
Photos of Test Setup:
These photos show the test setup on my desk. The Uno and connected hardware on one half of the breadboard are not being used.
I'm testing with just one slave at the moment. The issue I'm seeing is that the 'request' data packet that is sent from the master to the slave at 20Hz causes the I2C on the slave to hang. The slave hangs a large majority of the time in the loop() part of the sketch in calls which request data from the MPU6050 sensor, primarily 'mpu.getFIFOCount()' and 'mpu.getFIFOBytes'. Other times it hangs at startup, after a reset for example, on initializing the sensor during setup() such as 'mpu.dmpInitialize'. I've tried using the sensor without the built in DMP feature (so just reading raw accel and gyro values) and still get the hangs.
I'm using a timeout in the Wire library which I believe worked in a previous project but it is not 'timing out' the hang in this case. Whether this is because the timeout isn't implemented properly or whether the issue at hand is not related to the type of problem that the timeout solves I'm not sure. [Update: My Arduino IDE was not recompiling the modified timeout version of the Wire library when uploading to the Mega2560, that's why the timeout wasn't working. It's now fixed and the timeout works.] However the timeout is a band aid fix though and I2C hangs can generally be due to a hardware issue so I don't want to rely on the timeout, I want to find the underlying issue. Same goes for watchdog timers.
I'm viewing the I2C signals and the data requests on UART RX on the slave using an oscilloscope. If I disable the request signal, such as by turning off the master, the I2C runs fine indefinately. Also the slower I send the request the less often the hang.
Here are some oscilloscope screenshots showing the hang. The channels from top to bottom are SDA (yellow), SCL (light blue), UART RX (purple), UART TX (dark blue) on the slave. Note the oscilloscope mode I used allowed me to capture the hang (I don't have timeout trigger) but resulted in low resolution/sample-rate display, so some information may not be shown (noise/spikes).
There are others who have had the same problem with UART RX signals causing I2C hangs:
An Apparent Solution:
Something that seems to work is to use a resistor(am testing ~300 ohms) in series on SDA and SCL. I'm also trialing another resistor on the RX line (am testing ~1k ohms). This approach was taken from this thread here and also this webpage under the heading 'Don't "despike" your signal lines, add a resistor instead'. So far testing this approach allows the device to run for hours without hanging while accepting the request signal. It seems that the series resistor and the line capacitance creates a low pass filter of sorts and helps reduce noise on the line.
Understanding the Cause:
So the reason for this post is that I don't understand why this solution works so far and I want to discuss it here for my sake and for future enthusiasts who come across the same issue. This device will also be something that will be used by others and I don't feel comfortable moving forward with it without understanding the issue properly and ensuring it is solved properly.
Two of the oscilloscope screenshots show the SCL signal going LOW but stops before it reaches LOW and returns to HIGH. I have seen this many times when looking at these hangs on my oscilloscope, though it doesn't always happen. It's almost as if the Mega2560 releases the SCL whilst in the middle of pulling it LOW, because of the incoming signal on UART RX. I thought perhaps it's an interrupt issue but the series resistor solution seems to rule that out(?).
The series resistor solution seems to suggest noise being the issue, but then does that mean the RX signal is noisy? And if so why does a noisy UART RX signal hang the I2C? They must be electrically coupled somewhat?
I have tested leaving the RS485 transmit enabled on the master but with no actual request signal being sent to the slave to see if external EMI/RF noise is being coupled into the 3m CAT5E cable but I didn't get any hangs with this test.
Any input much appreciated.