Corrupted variables randomly

Hello!

I have made a mini bell system for home. I though I would come through cheaper if I make Arduino clones myself which share the same clock. One Arduino sends clock to the next. 5 of them are wired like this and it worked pretty well. Recently I changed the power supply from 5 cheap wall warts (one for arduinos, rest for motors) to one old PC power supply. The Arduinos were and are always supplied through 5V regulator (from 12V).
For easier changing of variables I added pc communication and variables are stored in eeprom.

After I changed to PC power supply, every once in a while one of them will start behaving strangely. Week ago one of them went crazy and started pulling the motor in only one direction never stopping (it should have been swinging). In past few days it always so happens that another of them stops responding to signals randomly during the day (usually only once). When I connected PC to it and asked for variables (to see if its still running) it sent me corrupted variables:

normal:

corrupted:

(explanation: power duty cycle and power iterations give the motor more power for the first few swings)

Now, I know the pc connection is working as I have used it many times in the past to set variables and if it wasnt there would be wrong variables stored on the Arduino in the first place. I also use CRC to check sent data.
When I reset the Arduino everything returns to normal. Variables are loaded from EEPROM so it must also be working properly.
Something happens during "waiting" time which corrupts the arduino is my guess. A few hours ago a hammer unexpectedly striked, and as I am using pull-up resistors it could be a sign that voltage has dropped for a fraction of a second, triggering the signal, and has also a chance of corrupting arduino memory (?) (note that I AM using decoupling caps on each of them).
I should also state that I am not using dummy load on power supply.

Did anyone have any similar problems and could help me with this? Much appreciated!

Here are some pictures of the setup:

entire setup
regulator and oscillator
control board
one unit
pc communication

In summary:
Separate Power Supplies is OK
One Common Power Supply FAILS

  1. Ground Loop ?
  2. Motor noise no longer isolated ?

mrsummitville:
In summary:
Separate Power Supplies is OK
One Common Power Supply FAILS

  1. Ground Loop ?
  2. Motor noise no longer isolated ?

Thank you for reply!

Yes, it looks like it. I didn't have those kinds of problems before and it ran for at least a month if not more.

  1. I have it wired up something like this:

Now I am not an expert, could you tell me if there is anything that might indicate potential ground loop in the picture?

  1. Hmm could be actually, I am using locked antiphase that has a lot of back surge current (as I read in an article about it). Can noise get through regulator? Although it still could be in inverse direction, from ground if any of the back surge current is getting out of h-bridge (thats me philosophizing, not sure if thats possible, it does have diodes on it, if they are responding to 20kHz pwm).

However I should note that when I turn the motors on and off (via command to arduino) to try to duplicate the problem it doesnt happen. It seems to happen sometime during the day while motors are still (and arduinos are waiting), and when its time to work one of them doesnt. I would blame it on defective chip if it didnt already happen on one of the others (which went crazy, and was then fine after reset).

I have connected the regulator now to Mega's VIN and im interested to know if the corruption will stop.
Not being able to repeat the problem manually is really annoying.

One thing I forgot to mention is that I also changed Arduino Uno with Arduino Mega the same time I changed the power supply, but the code is the same so I don't see how this could be causing these issues.

Okay the problem has escalated today... multiple of them are weird. It seems the program is still running as some of them are responding to my commands, but they are crazy, instead of pulling motor gently it pull with full power. Two arent responsive and third is only pulling in one direction. What the heck o.O And it happened recently out of nowhere, 1 hour ago they were all working normally.

Update: I have tried pulling the regulator 7805's power out (shut it down) and to my surprise the ATmegas were STILL WORKING! After trying to pull out a few connections, I found out that they shut down if I sever the connection to Chinese Arduino Mega. They are being supplied through Mega's I/O pins somehow. I dont know why I started having problems with only this Arduino Mega as I used the same principle before with Arduino Uno, even the pin numbers are the same.

I am sorry that no-one has addressed this for you in the last three days, because it is so blatantly obvious.

Your whole mock-up is guaranteed to be a total disaster.

Your concept of using a common clock is the first problem. I cannot quite figure out what the supposed benefit was, but simply - forget it! You need to provide a clock crystal for each microprocessor. Not only must you provide it, but it must be close to the processor, generally within 2 cm, preferably half that. Proper clock design (probably mentioned in the datasheet, but I can't be bothered finding it) dictates that there be a ground "guard" trace surrounding the crystal and loading capacitor connections.

I do note your attention to the decoupling capacitors and I hope you have the ground and power bridged between each side of the chip under the board, but given the other problems, this is negligible reassurance.

The other outstanding problem is the "rats nest" wiring. It is frankly unimaginable that this could ever function at the frequencies and impedance levels involved in microprocessors. All your complaints are in this regard, entirely predictable and it is a tribute to Heath Robinson that it might have even functioned once.

You need to re-think the entire project - at least in the hardware side. I can only suggest you start by going and purchasing a dozen or so of the cheap Pro Minis available on eBay such as from this seller whom many of us have found reliable. Then lay them out carefully, using ribbon cable (and for re-work, "Dupont" connectors with multi-pin housings) to run from one to the next, or from each module to its corresponding sensors. The rule is - all interconnections are paired, supply lines together and sensor lines together, all together wherever practical.

Paul__B:
I am sorry that no-one has addressed this for you in the last three days, because it is so blatantly obvious.

Your whole mock-up is guaranteed to be a total disaster.

Your concept of using a common clock is the first problem. I cannot quite figure out what the supposed benefit was, but simply - forget it! You need to provide a clock crystal for each microprocessor. Not only must you provide it, but it must be close to the processor, generally within 2 cm, preferably half that. Proper clock design (probably mentioned in the datasheet, but I can't be bothered finding it) dictates that there be a ground "guard" trace surrounding the crystal and loading capacitor connections.

I do note your attention to the decoupling capacitors and I hope you have the ground and power bridged between each side of the chip under the board, but given the other problems, this is negligible reassurance.

The other outstanding problem is the "rats nest" wiring. It is frankly unimaginable that this could ever function at the frequencies and impedance levels involved in microprocessors. All your complaints are in this regard, entirely predictable and it is a tribute to Heath Robinson that it might have even functioned once.

You need to re-think the entire project - at least in the hardware side. I can only suggest you start by going and purchasing a dozen or so of the cheap Pro Minis available on eBay such as from this seller whom many of us have found reliable. Then lay them out carefully, using ribbon cable (and for re-work, "Dupont" connectors with multi-pin housings) to run from one to the next, or from each module to its corresponding sensors. The rule is - all interconnections are paired, supply lines together and sensor lines together, all together wherever practical.

You are, ofcourse, right. It does look like rat's nest and even I wasn't sure it was going to work. It was the first a bit bigger project with electronics that I started making and I learned and modified things as I got along. I was shaping the idea in my mind and wanted to test it as cheaply as possible. I didn't know that I could get stuff on ebay so cheaply when I bought the MCUs and FETs (I even wanted to build the h-bridges manually, but fortunately I found out about ebay stuff). Since I already had the MCUs and wanted to put them to use I built these boards. I didn't even imagine how time consuming it would be. Im still modifying and adding stuff. I like your idea about the Minis and now, when its almost complete, I might migrate to a more reliable and clean setup. I hate the wiring and really want to do something about it, I just haven't gotten about ordering the cables from ebay yet (as I don't know where to get them in my country exactly and didnt know how many and which ones should I get).

The idea behind the clock doesn't come from lower price though, it comes from desire for MCUs to never get out of sync. The clock isnt connected to all of them directly, each MCU is confgured so that it passes the clock signal to the next one. My fear of desync was probably exaggerated, I must admit. Its a 4-pin oscillator module though, and it already contains all necessary parts (as I read on the internet).

The actual problem though was that I connected two different mcu's to different power sources, and then they were talking to each other via signals. I was unaware of the internal clamping diode. For now, its again working normally (I connected both MCUs to same power source and I have to buy a data only USB cable to keep using USB on Mega), and the only way the problem recurred is by connecting solenoid without diodes to the same power supply.

Thanks for your suggestions and have a nice day!

The idea behind the clock doesn't come from lower price though, it comes from desire for MCUs to never get out of sync. The clock isnt connected to all of them directly, each MCU is confgured so that it passes the clock signal to the next one. My fear of desync was probably exaggerated, I must admit. Its a 4-pin oscillator module though, and it already contains all necessary parts (as I read on the internet).

A very interresting project.
I like digital clocks, but I am anal regarding their time. The solution was the GPS clock because here in the Southern U.S. the RF based WWVH clocks only receive a sync signal irregularly. But, even at the Chinese price of $11 each, the GPS modules add up. My solution was one GPS base unit in my attic and an ESP8266 programmed via the ArduinoIDE to broadcast the time/date via WiFi to UDP listeners:

Tardis Time

Ray