Random crashes, suspect is power in the building

Hello, I think this is the right place to post this kind of a question but if not please redirect to the more appropriate forum.

The home base of the project is in a college warehouse with my boss, while I have been working remotely to provide updated code on an identical hardware configuration.

So we've been having some random crashes with the our arduino's, where they freeze and get stuck as opposed to a complete crash with a reboot which would be fine, and haven't been able to nail down where the issues are coming from. I've eliminated our software and hardware as the cause for the following troubleshot reasons.

  • Once we realized the crashing was happening at the warehouse I tried running the same code remotely and it worked until unplugging it manually.
  • I gradually started eliminating sections/features of the code corresponding to different hardware pieces until all I was left with was a 2x16 lcd display blinking a static string. The warehouse arduino's were stilling crashing (every few hours but not on a predictable schedule) but my home setup was working fine.

Best guess I have now is that there might be something off about the power in building that's causing the arduino's to get stuck. However this doesn't quite make sense because we have another environmental system, commercial, as a place holder while we get ours working and it doesn't experience this issue.

Could there potentially be some dips in the current provided short enough to cause uno's to get stuck while leaving other devices operational, like when the lights suddenly go dim when someone plugs in a kettle only not apparently correspondent with other equipment? If so how would I monitor for this?

I picked up a UPS battery back up unit with a transfer time of 6ms to try and tell if my theory is correct if it can pick up the slack.

Would any of you have any insight?

If you are using String objects on a standard Arduino, that would also be a suspect. Post the code, using code tags.

Lots of text but no real information, no code, no wiring, no set up. Try again.

Is your hardware setup identical to the equipment at the warehouse? Are the inputs/outputs identical? Seems unlikely to be code if the hardware is truly identical, unless the warehouse equipment is having to process input data that you are not correctly simulating. Can the arduino at the warehouse be run off battery for an extended time period to see if that eliminates the crashing?

Does the warehouse have any sources of electrical or RF noise that might be getting picked up in the wiring of the arduino and peripherals?

@david_2018

Yes they are truly identical. This reminded me, forgot to mention in the OP, but my boss brought down one of the setups to my place, and I tested it there, no alterations, and it worked just fine for several days, again until I unplugged it.

As far as input data, temperature relative humidity and CO2ppm, are all being simulated to the same levels.

With respect to electrical noise, I was worried about this too except for the fact that some of these crashes happen over night when the equipment capable of generating any ambient noise are not running. We have a 480V three phase motor, a boiler, and a glycerol cooler.

Are You using any breadboard?

@railroader

On one of four of our setups yes. But the other are all soldered or clips (ethernet)

AustinMorris:
@david_2018

Yes they are truly identical. This reminded me, forgot to mention in the OP, but my boss brought down one of the setups to my place, and I tested it there, no alterations, and it worked just fine for several days, again until I unplugged it.

As far as input data, temperature relative humidity and CO2ppm, are all being simulated to the same levels.

With respect to electrical noise, I was worried about this too except for the fact that some of these crashes happen over night when the equipment capable of generating any ambient noise are not running. We have a 480V three phase motor, a boiler, and a glycerol cooler.

What other buildings share the 480 volt, 3-phase power? The problem may be not in your building.

You can also apply to your power supplier to use their power monitoring equipment to document power problems. Then it will be up to them to correct it.

Paul

Paul_KD7HB:
What other buildings share the 480 volt, 3-phase power? The problem may be not in your building.

You can also apply to your power supplier to use their power monitoring equipment to document power problems. Then it will be up to them to correct it.

Paul

Good to know! I'll inquire about that applying to our utility company provided more troubleshooting points in that direction.

Unfortunately I don't know exactly who else would. Our college campus is a rural focused school, brewmasters, greenhouses, everything you can imagine rural focused. When we got the mixer in that uses the 480V 3 phase power we had to have the line installed in the warehouse. Whether we got a new line direct from the transformer or one branched off from another line I'm unsure and all staff are still out until next week.

I have experinced that sometimes those breaboards and the lab-cables give a very unreliable contact. Any little vibration or movement causes hanging, nonprintable characters in my I2C LCD etc..
Can You solder the breadboard stuff to make shure Your project don't suffer from such trouble?

AustinMorris:
Good to know! I'll inquire about that applying to our utility company provided more troubleshooting points in that direction.

Unfortunately I don't know exactly who else would. Our college campus is a rural focused school, brewmasters, greenhouses, everything you can imagine rural focused. When we got the mixer in that uses the 480V 3 phase power we had to have the line installed in the warehouse. Whether we got a new line direct from the transformer or one branched off from another line I'm unsure and all staff are still out until next week.

If the 480 volt comes overhead, then there will be a common pole with three transformers. One for each phase. If the power is underground, then you still have 3 transformers, but they will be enclosed in a single big steel box, usually green in color.

Paul

"the lights suddenly go dim when someone plugs in a kettle"

That to me would mean a voltage sag. Do you have a DMM with Max/Min function? You could test with that when operating the kettle or other loads and see if there are any sags and/or spikes. There needs to be a step down transformer somewhere. It may not be sized correctly, if a lot of low voltage circuits were added. You may also have a problem with grounding or bonding of the transformer, causing a floating neutral.

Ron