Yun Randomly hanging

OK so the Yun has been up for more than 2 days now. So it seems it was a faulty sensor or wire. I got the sensor back from the unit in the field. I will be testing it with the dev unit. One more question is if the Arduino is only for prototyping, what is the full production unit then ? Or is Arduino only seen as a toy ?

jfourie:
One more question is if the Arduino is only for prototyping, what is the full production unit then ? Or is Arduino only seen as a toy ?

That's an interesting philosophical question, one which I don't think has a black and white answer: it is what you want it to be, so the answer is really different shades of grey, depending on who you are and who you ask.

I think the Arduino platform is targeted at the beginner/hobbyist/artist, and I don't think it's taken seriously in the professional/industrial world. There's nothing wrong with that: it has a stated purpose, and it fulfills that purpose admirably. It has brought the world of embedded electronics and programming to a group of people that haven't had access to it before. Traditionally, the barriers to get into embedded programming are high (expensive and/or sophisticated development environments, working with bare processors and electronics, etc.) but the Arduino environment/culture has significantly lowered the barriers and made the technology much more easily approachable.

There is nothing in the Arduino design that precludes it from being used in a production environment. However, using it as a prototyping platform, and then designing a custom built product has certain advantages: while the cost of most Arduino boards is low, the individual parts are cheaper, and if you are making more than a few of your product, you can probably build a custom board that includes the processor and all of your task-specific hardware that will cost less than just building a shield for your custom hardware and using an off-the shelf processor board. In addition, a custom solution may let you design something that is more robust (better power supplies, better input/output conditioning, etc.)

I think it's mostly a matter of the economies of scale: build just a few units, and it's probably cheaper to use an available preexisting solution (like an Arduino processor board) because the engineering expense of designing a board from scratch can be expensive compared to the cost of a few commercial boards. But if you are building many thousands of units, the the non-recurring engineering expense is spread out over a greater number of boards, and the engineering cost per board drops dramatically. At that point, the cost of the parts becomes significant, and it can be much cheaper to buy and assemble the individual parts than it is to purchase complete boards. Given high enough production quantities, saving just a few cents per board can quickly multiply into significant additional profits.

In my professional career, where we design and program custom hardware according to a customer's specifications, we would never use a prototyping platform as a basis for the final product - it's always a full custom development. Of course, we will sometimes use a development board like a Texas Instruments LauchPad board (ultra low power MSP430 processor) or a Freescale (now NXP) Tower System (high performance Kinetis ARM RISC processor) for the early development stages, but that is typically only for a proof of concept, or to have a platform to start coding on while waiting for the hardware to be designed and built.

However, in my personal life, where I'm working on mostly one-off projects, time and money are more important, and I will generally incorporate the Arduino board in the finished project - it just doesn't make sense to spend all of that time recreating something that already works. This is especially true of my Yun based projects, since the underlying hardware is so complicated to duplicate.

So it's a tough call, one that requires a personal decision. In the commercial world, besides the benefits of economy of scale and the ability to customize the processor to the environment, there is also the appearance of professionalism - Arduino is often viewed as a hobbyist toy, and including one in a commercial product may be seen to diminish the stature of the product. Also, "Not Invented Here" could be a factor. However, there really isn't anything in the Arduino that makes it generally unsuitable in a production environment - as long as it meets your needs, and works in the operational conditions, there is no reason to exclude it just because it's primarily intended as a prototyping platform.

Thanks for your reply. We are probably going off topic here but it is still an important question to answer. I have identified a certain group of people who I want to help by automating a lot of what they currently do by hand.

The Arduino platform is affordable in that space and will work 100% if it is stable and durable. So it won’t be a 1000 units of the same thing. More 1000 units of different things in once off situations. I have a lot of experience in systems design as in my day job I am a solutions architect using Oracle and other high level programming languages like java and dot net. I also know C and C++. But my electronics knowledge is lacking. I am a fast learner a will if I have to learn that part. But I see the hardware as one tool in the toolbox. As long as the tool will work then I will use it.

Industrial grade PLC are too expensive for the market I want to target and help. But then the platform must be reliable. The fact that you get Industrial grade Arduino PLCs tell me it should be possible to use in for this reason or am I smoking my socks?

jfourie:
We are probably going off topic here but it is still a important question to answer.

Hey, it's your thread, you have the privilege of steering its course. 8)

As long as the tool will work then I will use it.

I agree, if the tool works, and does what you need it to do, with the reliability that you need, then use it. Only you can decide if the tool meets your criteria.

The fact that you get Industrial grade Arduino PLCs tell me it should be possible to use in for this reason or am I smoking my socks?

I'm not aware of industrial grade Arduino's, but then I don't really have any direct experience with PLCs. My only exposure to PLCs is when a customer has a machine they are building that is currently controlled by a PLC, but the PLC is making their product too expensive. In that case, we don't really worry about how the PLC is currently being used, we concentrate on what the machine should do, how it does it, and what the customer wants it to do. We then design a custom board to handle all of the required I/O and processing, and write custom software that meets the customer's needs. It's never the situation where they want a drop-in replacement that does only what the PLC does: there are always additional things they want it to do, things that were too difficult or expensive to do with the PLC. We've never done a board with an Atmel ATmega processor, only because we are not really familiar with it. Although, actually that's not true: we did build two copies of a custom board as pro-bono work to mentor a high school student with a technology project - we used a '32U4 and put an Arduino LilyPad bootloader on it, just so it could be programmed using the Arduino IDE. This was his first ever programming project, and I felt that was the best way to get him up to speed. The project was a resounding success.

Here are the Arduino that you can use in a industrial process.

So it seems that if you get these types of modules then I should be able to use them as I want to.

I need help please please please !!!!
The Yun keeps on hanging from time tot time.

What am I missing ? All works well and then for no reason at all it will just not respond.

I is not the wifi it has a good connection. The Yun was up for about 4 days when the drayer was off. Now the dryer is on and the yun started to hang again.

Could there be interference from cables or relays ? I don't know where else to look. We are now disconnecting all the sensors and let it run on its own. then will add one sensor at a time.

What is the most common thing that will let the controller hang ??

You've got a difficult situation on your hands, I don't envy you.

jfourie:
What is the most common thing that will let the controller hang ??

I believe that the most common thing that causes problems is an error in the program logic that doesn't handle an unexpected condition. This then causes a crash due to running out of memory, following a bad pointer, blowing the stack and returning to an unexpected location, etc. Unfortunately, that doesn't really help you.

About the only suggestion I have at this point is a serious debugging session, but it sounds like you're already heading that way. You're disconnecting sensors, and that may help you determine if an unexpected value or sensor error condition is causing your problem. In addition, I would suggest manually walking through your code line-by-line and think about the consequences of each action. For example, if reading in data from a sensor, what happens if the sensor returns an out of bound condition? Could you get an overflow in your calculations, if so, what would happen? Could you get a divide by zero? Could you overrun a buffer? Could you leak memory? And on and on... When reading some a sensor, what happens if it stops responding in the middle of a transaction? What happens if there is noise glitches on the data lines that cause unexpected states?

Unfortunately, the possibilities are pretty much endless.

OK let say it is a program error. Should the watchdog not restart the controller ? It hangs to a state that you have to cycle the power.
What good is a watchdog if it does not reset the controller on a software hang.

jfourie:
Should the watchdog not restart the controller ?

Is the watchdog not restarting the controller? Or is it restarting the controller but not resolving the problem? There is a significant difference there.

It hangs to a state that you have to cycle the power.

If it is the sketch in the '32U4 processor that is hanging, and the watchdog does fire and reset the controller, but the system still locks up, then that points to one of the I2C sensors being hung up and locking up the bus. In that case, the sketch may be restarting, and hanging again on the first I2C access. A simple software reset of the '32U4 processor will not clear that, the sensor itself would need to be reset.

That's assuming that it's the sketch that is hanging. If the problem is on the Linux side, then the '32U4 watchdog won't help, as there is no existing provision for the '32U4 to reset the Linux side. A Linux hang would have to be reset by a Linux watchdog. I looked into the Linux watchdog some more, and could not figure out how it would be useful to handle the situation where user process has hung. My hope was that the user process could open /dev/watchdog and periodically write to it to reset it. If the user process stopped writing to it, the system would reset. But it doesn't appear that such a scenario will work: my attempts to do that resulted in an open failure on /dev/watchdog, with an error message indicating that the file is busy.

As near as I can tell at this point, the Linux watchdog is only useful for monitoring key system processes. If you have come up with a way to make the Linux watchdog monitor your user task, I would be very interested in hearing it.

While it doesn't address the root cause, maybe the ultimate safety net is to use an external hardware watchdog like you previously mentioned? Set up a device with two service inputs: one controlled by the sketch on the '32U4 processor, and one by your Linux user process. (It is possible to set up the Linux side to control the digital pin 7 on the shield connector, and you could use that as your Linux output.) If either input is not pulsed for a period of time, the hardware would power cycle the system. This could be a dedicated chip, or could be another Arduino like an Uno, Mini, or Micro. If using another Arduino, you just need to make sure that it also has its internal watchdog enabled.

Finally, however you reset the system (internal watchdog, external watchdog, or manually cycling power) there is the possibility that the Linux software could hang on booting. In my first reply to this thread, I provided a link on how to improve reboot stability. Have you taken any of those steps to make sure that isn't contributing to your issues?

This is a rather one-sided conversation... I'm surprised that nobody else has stepped in to offer some suggestions?

Hi, Yes I did do as stated in the link. The yun will now reboot and will come back to a working state if I use the command reboot. Maybe the yun is not the tool for the job. Maybe I should use someting like a Pine 64 or a RPi and use the a Ardiono Uno to connect to the sensors. I see you get the libraries for Python so Maybe I should just use a RPi and test it to see if it works and stays up.

Can it be the iphone charger I use as a power supply ?

OK so I think I have the problem solved (I hope). It was never a hang. It was power loss. Because the controller is hooked up in an industrial environment I wanted to shield it from power spikes. So I added a surge protector multi plug. As it turns out this plug does not fit into the socket 100%. So when the dryer starts and there is vibration the plug will rattle out of the socket and then power off the controller. I picked this up when the UPS that I asked them to add also died without power. And the LCD was dead every time they told me it hanged (it was dead not frozen). It was very difficult to find this problem as I am 200km from the controller and the information the people on the other end gave me led me on a wild goose chase.

Good thing is that now I will have external watchdogs in place to make sure all stays the the way it should do.

Shapeshfiter thanks for all your help and info. Let hope all stays up and working now :wink:

Regards.

Jaco.

jfourie:
It was very difficult to find this problem as I am 200km from the controller and the information the people on the other end gave me led me on a wild goose chase.

Yes, remote debugging can be the worst thing, especially when you can't rely on the people to give you good information.

I've been there and done that, and it's no fun. At least you didn't have to get on a plane to go there just to find out it's something stupid: true story, years ago -- the boss is at a remote customer site complaining that communications would some times lock up. I look at the code I inherited, and quickly find a bug that could cause what he's describing. I explain the failure mode to him, and how it can be triggered by a simple operator action. He emphatically says that's not what was happening, so I load up a bunch of computers, test equipment, etc. and get on a plane. This was 30 years ago: it wasn't just a laptop with a USB programming pod, it was a desktop computer chassis, CRT monitor, EEPROM programmer, serial line protocol analyzer, and more (they rented me a van to get it all from the airport to the customer site.) So here I am working outside for a few days with equipment open and my test equipment hanging all over the place, and I just can't get it to fail. The ONLY way I could get it to fail is the scenario I originally discovered. So I finally show that failure scenario to the boss, the scenario I described over the phone, and he immediately says "Yes, that's what we were doing." WHAT?!?!?! If he had just admitted to that over the phone, I could've fixed it in 15 minutes without ever leaving the office. Instead, I spent four days and who knows how many dollars in other costs (airfare, hotel, shipping, etc.) to make the same fix. With management like that, it's no wonder that the company went out of business.

Oh well, hopefully this will solve your problems and it will be smooth sailing from now on. I guess that's why the tech support cliche is to ask if it's plugged in as the first step?