i2c hanging while motors running

Using Arduino Networking, Protocols, and Devices
#1

i'm finding that calls to I2C routines are not returning while testing code to control model railroad tortoise switch machines. The arduino controls boards with an MPC23017 and LM324 op-amps. Two op-amp outputs control the polarity to a stall motor in the switch machine that consumes < 20 ma. power and I2C connections are daisy chained between boards.

3731×1567 1.17 MB

i've been able to isolate the problem to I2C routine calls. i've further isolated the cause by avoiding any I2C operations by waiting some period (4 sec) for all motors to reach their travel limits.

I see an infinite loop in twi_readFrom()

while(TWI_READY != twi_state){
continue;
}

as an embedded firmware developer, we would never write such a loop without some timeout/count-out mechanism.

i've seen another thread from 2011 that never discussed a solution but mentioned noise as a potential cause. I realize the way I wired the board may be an issue and the I2C wiring and power wiring easily separated from one another.

of course i'm interested in a software fix, but interested in better understanding how noise could be the problem.

#2

gciurpita:
as an embedded firmware developer, we would never write such a loop without some timeout/count-out mechanism.

When you shortcut SDA or SCL to GND, then the Wire library halts, and the sketch does no longer run. An event from the outside can stop the Arduino. That is terrible and very stupid. I'm waiting for many years and hoping that the Arduino team someday will add timeouts to the Wire library. Arduino was developed at a university and not by embedded developers with many years of experience.

If you want to use an Arduino board in a serious project, then you should use the Atmel Studio with the libraries that comes with that IDE.

Arduino is for hobby and fast prototyping. Because of the Wire library without timeouts, it is still in its infant state.

You should seperate the GND wires for the I2C and the GND wires for the motors. I don't know if your driver chips can seperate those. The I2C bus has three wires: SDA, SCL, GND. It was designed to be used on the same pcb board.

Without external pullup resistors, only the internal pullup resistors of the Arduino Uno are used. They are too weak, and you get all kinds of electrical noise. Often 4k7 pullups are used, but you can try 2k2 to start with. That is 2k2 from SDA to 5V and 2k2 from SCL to 5V.

Try lowering the clock speed to 50k with Wire.setClock( 50000) after Wire.begin().

Try the Multispeed Scanner: MultiSpeed I2C Scanner - 50,100,200,400 KHz. - Libraries - Arduino Forum.
If it works at 400kHz, then 100kHz is okay.

That cheap DC/DC-converter is noisy. When the power and GND wires (with current from the motors and noise from the DC/DC-converter) are near SDA and SCL, you are injecting noise into the I2C bus. Can you wire the power wires at the top and the data wires at the bottom ?

Where is the Arduino GND connected to the DC/DC-converter GND ? Perhaps a single GND point near the Arduino board is the best (Star Ground). No cascading, every module and also the DC/DC-converter GND to that GND point.

#3

I like your breadboard, I have seen this before on my own systems. I traced it down to the libraries as stated above because if they do not get a response they just wait for one. That puts you in do nothing mode. I found it by putting a print statement before and after each wire access, it became obvious very quickly. From your pictures I do not see bypass capacitor nor do I see any bulk capacitance If I am correct this can lead to the problems you are seeing. I also assume the boards are being powered by a power source not sourced by the Arduino which is correct. Double check the grounds they like playing games. Another thing you can do is to slow down the I2C speed, this helps reduce noise problems. You can also check the I2C pull up resistors, A scope would be great. Try lowering them on the end board on the bus.
Good Luck & Have Fun!
Gil

#4

thanks

i've already found that the wiring makes a difference. I found that a star configuration (each board in parallel) is worst.

i'll try the pull-ups

wondering why not directly use the twi routines instead of the Wire functions

#5

The I2C bus requires pullup resistors, or it won't work at all. With the weak internal pullup resistors it will work barely or not.

The Arduino Wire library is what every other 'Wire' library tries to be compatible with.

The Arduino Wire library for the Arduino Uno happens to have a low level layer. Some are tempted to "improve" the Wire library and some are tempted to use the low level functions. So far, everyone has failed and has gotten into more trouble. You are no exception to that. You can not get a single benefit from the low level functions.
There was even someone that could not make his I2C bus work and it turned out that he used his own "improved" version of the Wire library, which causes all the problems :smiling_imp:

#6

what i'm curious about is what exactly is failing and why do the library routines hang?

i understand that noise can corrupt both the clock or data.

if the address data is corrupt, i would not expect an acknowledgement. I would expect the code to timeout and return an error.

If the address is received correctly, I wouldn't be surprised in the clock is corrupted, resulting in too few and the receiver not acknowledging while waiting for the final clock. Again, I would expect the code to timeout and return an error.

i can understand that the data is received incorrectly. I don't understand why the code should hang.

I've had to investigate problems like this with incredibly low probabilities that required stress testing and overnight tests to reproduce. that is why a timeout is almost always required in loop.

#7

Sorry, I don't know. If the SDA or SCL is shortcut to GND or to each other, then the Wire library keeps on waiting for a bit in a register. A normal sensor does not cause a shortcut. I assume that there is some kind of lockup. I'm not sure if I ever have seen a bus capture picture of that, I don't think so.
The I2C protocol is pretty good. A sensor can probably detect a START or STOP condition regardless in what state it is. That means that if the Wire library should continue, then it is probably able to communicate with the sensor again.

gciurpita:
that is why a timeout is almost always required in loop.

Do you want me to rant again about the Wire library ? We have already establed that it is terribly stupid.

#8

i've seen no hangs since adding two 4.7K pull-up resistors to the end of the SDA and SCL buses. Clearly an oversite on my part. I think this makes a lot of sense since without sufficient pull-ups, both buses float and are very susceptible to corruption by noise.

thanks for pointing out their need

but it's still worrisome that the code is susceptible to hang if there is corruption.

Koepel:
Do you want me to rant again about the Wire library ? We have already establed that it is terribly stupid.

i'd be interested in reading about specific weaknesses (specific lines of code) of the Wire library. Shouldn't it be easy to add timeouts to loops to at least prevent it from ever hanging.

Koepel:
If you want to use an Arduino board in a serious project, then you should use the Atmel Studio with the libraries that comes with that IDE.

as far as I know, the usb interface to programming an Arduino is rather unique. how easy would it be to use the Atmel studio to build code that can easily be downloaded thru USB and does it still support use of the Arduino IDE serial monitor.

#9

The timeouts of the Wire library is a big subject.

Some make a Wire library with timeouts, but do not maintain their library. When a bug is fixed in the Arduino Wire library, they do not update their library. So I can't use such a library either.
The Wire library is part of the Arduino Stream family and thus the setTimeout() function can be used. However that is at a higher level, a low level timeout for the I2C bus is something else.
There are issues and pull requests, but the Wire library is not so easy, and the Arduino team does not care about it.
Some make their own version, and decide to make it non-blocking as well. But then they use the same function names. That is wrong, because then it is no longer compatible.

Even the most simple thing, using the Wire library, is already a problem for many. There is even someone who made it his personal goal to tell others to use the Wire library in the right way.

I continue to use the Arduino Wire library and make sure that my I2C bus is working well. I think it is up to the Arduino team to make a Wire library with timeouts.
If you want to fix it, then you do what so many have done before you. You can start with all the while-statements here: ArduinoCore-avr/twi.c at master · arduino/ArduinoCore-avr · GitHub.

Do you have 2k2 resistors ? The maximum current to pull a signal low is 3mA for a normal I2C bus. 5V / 3mA = 1667 Ω. So all your pullup resistors combined (inclusive the internal pullup resistors) should not be lower than 1667 Ω.

#10