Linino Locks up after ~12-24 hours Running php Script every minute

I have a project that logs data from Arduino to a sqlite3 database on Linino: http://forum.arduino.cc/index.php?topic=227344.0

I have solved some issues with help from people here, but am now experiencing a recurring problem. I run a php script on Linino via a cronjob that runs every minute. This php script syncs data from the local sqlite3 database to a remote MySQL database for web access. The problem is that after about 12 hours (almost exactly on some occasions), but sometimes after 24 hours, the dbSync.php script no longer runs (the data doesn't make it to my webserver). In the meantime, it appears that the sqlite3 command run through the bridge still works. After I reset Linino I see that all the data during the time when the php script was not running was saved locally. After this reboot the script syncs all the missing data.

Furthermore, I can still see the Yun on my router, but cannot access it via HTTP config access or SSH. Any idea what might be going on here? It appears parts of Linino become non-functional, but that the OS is still running somehow (since it can execute shell commands via the bridge). I suspect I will get asked for the PHP script, which I will provide in the morning. In the meantime, any ideas? Since this is a remote application, are there any other suggestions on how I can try to communicate with/reset the Yun remotely?

Thanks

Nagios (opensource) is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes. Nagios is installed a Linux host machine.

  • Monitoring
  • Alerting
  • Response
  • Reporting
  • Maintenance
  • Planning

NRPE - Nagios Remote Plugin Executor, load NRPE at Yun that allow you monitor network connection, cpu usage, memory usage…, and send out alert via email or sms once threshold is reached.

Setting up Nagios to remotely monitor an openwrt router ( Arduino Yun)

sonnyyu: NRPE - Nagios Remote Plugin Executor, load NRPE at Yun that allow you monitor network connection, cpu usage, memory usage..., and send out alert via email or sms once threshold is reached.

Setting up Nagios to remotely monitor an openwrt router ( Arduino Yun)

Thanks, this is a great suggestion. I will get it up and running. In the meantime, any suggestions on how to access or reboot my partially locked up Yun that is in the field?

Re: Measuring if the Yun is internet connected

http://forum.arduino.cc/index.php?topic=209412.msg1541242#msg1541242

Modify the code and make it run once only and reboot linino when no internet. then put it at cronjob that runs every ~10 minutes.

B.T.W. Nagios support response which means once no network then auto reboot.

sonnyyu: NRPE - Nagios Remote Plugin Executor, load NRPE at Yun that allow you monitor network connection, cpu usage, memory usage..., and send out alert via email or sms once threshold is reached.

Setting up Nagios to remotely monitor an openwrt router ( Arduino Yun)

I have tried installing this both from Luci and from SSH command line and cannot, for some strange reason. In Luci I was able to Update Packages and install 'nrpe' package, but not 'nagios-plugins'. I can view the 'nagios-plugins' package when I search for it in Luci, but when I try to install I get a 'Bad Gateway - The process did not produce any response' page after timing out. I should mention that during the process of installing, my Yun lost internet connection, so I checked 'netstat -nr' and added '/sbin/rout add -net 0.0.0.0 gw 192.168.1.1 eth1'. My Yun is now able to connect to internet and obtain package list, but I cannot install anything. Any idea what went wrong? This is frustrating because it is diverting me from the initial issue of this post. Thanks for the help!

root@yun01:~# netstat -n -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
192.168.240.0   0.0.0.0         255.255.255.0   U         0 0          0 wlan0


but when I try to install I get a ‘Bad Gateway - The process did not produce any response’ page after timing out. I should mention that during the process of installing, my Yun lost internet connection, so I checked ‘netstat -nr’ and added ‘/sbin/rout add -net 0.0.0.0 gw 192.168.1.1 eth1’. My Yun is now able to connect to internet and obtain package list, but I cannot install anything. Any idea what went wrong? This is frustrating because it is diverting me from the initial issue of this post.

DHCP patch is needed.

sonnyyu:

... but when I try to install I get a 'Bad Gateway - The process did not produce any response' page after timing out. I should mention that during the process of installing, my Yun lost internet connection, so I checked 'netstat -nr' and added '/sbin/rout add -net 0.0.0.0 gw 192.168.1.1 eth1'. My Yun is now able to connect to internet and obtain package list, but I cannot install anything. Any idea what went wrong? This is frustrating because it is diverting me from the initial issue of this post. ...

http://forum.arduino.cc/index.php?topic=203754.msg1507343#msg1507343

DHCP patch is needed.

Even if I have a static IP setup in the router I hook the Yun to with eth1? And even if I am still able to ping other web locations?

whisprer:

sonnyyu:

... but when I try to install I get a 'Bad Gateway - The process did not produce any response' page after timing out. I should mention that during the process of installing, my Yun lost internet connection, so I checked 'netstat -nr' and added '/sbin/rout add -net 0.0.0.0 gw 192.168.1.1 eth1'. My Yun is now able to connect to internet and obtain package list, but I cannot install anything. Any idea what went wrong? This is frustrating because it is diverting me from the initial issue of this post. ...

http://forum.arduino.cc/index.php?topic=203754.msg1507343#msg1507343

DHCP patch is needed.

Even if I have a static IP setup in the router I hook the Yun to with eth1? And even if I am still able to ping other web locations?

I implemented the fix you linked to and it did nothing to solve the problem. I essentially implemented this manually via SSH already. On top of this, I haven't had any issues over the past months with this before.

These shell commands work sometimes, but are flaky. I cannot install nagios-plugins.

opkg update
and 
opkg install

I cannot update packages from Luci or install anything (Bad Gateway). I have restored the Yun to factory defaults and implemented your fix, and it still does not correct the issue. There is either something seriously wrong with my Yun now or something wrong with the linino server. Any insights? Thanks!

Even if I have a static IP setup in the router I hook the Yun to with eth1? And even if I am still able to ping other web locations?

No need DHCP patch for static IP , post the output of

cat /etc/config/network

Start from fresh (backup your own file!):

Code:

/usr/bin/reset-to-factory-anyway
reboot

sonnyyu:

Even if I have a static IP setup in the router I hook the Yun to with eth1? And even if I am still able to ping other web locations?

No need DHCP patch for static IP , post the output of

cat /etc/config/network
root@yun01:~# cat /etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config interface 'lan'
        option proto 'static'
        option ipaddr '192.168.240.1'
        option netmask '255.255.255.0'

config interface 'wan'
        option ifname 'eth1'
        option proto 'dhcp'
        option metric '10'

sonnyyu: Start from fresh (backup your own file!):

Code:

/usr/bin/reset-to-factory-anyway
reboot

Is this the exact same as holding the WLAN RST button for >30 seconds?

whisprer:

sonnyyu: Start from fresh (backup your own file!):

Code:

/usr/bin/reset-to-factory-anyway
reboot

Is this the exact same as holding the WLAN RST button for >30 seconds?

Same, but software is the way I prefer.

config interface 'wan' option ifname 'eth1' option proto 'dhcp' option metric '10'

You are on DHCP, the patch might help.

sonnyyu:

config interface 'wan' option ifname 'eth1' option proto 'dhcp' option metric '10'

You are on DHCP, the patch might help.

Yes, I set up the router give the Yun a static IP on it's network, but didn't change anything within the Yun's network settings. I have applied the fix anyways, even before my previous 2 posts, and I still cannot install packages. I am able to update the package list sometimes, but cannot install nagios-plugins or other packages I try. At one point I was luckily able to install nano. Any idea why I can no longer install packages? (Bad Gateway - the process did not produce any response)

post

ping sun.com
ping download.linino.org

reboot/power cycle wifi router, power cycle ethernet switch if you have one, power cycle ISP modem as well. It might be internet problem.

if you are working on 7/24 project then you need industrial grade router, switch as well as modem.

sonnyyu: post

ping sun.com
ping download.linino.org

reboot/power cycle wifi router, power cycle ethernet switch if you have one, power cycle ISP modem as well. It might be internet problem. if you are working on 7/24 project then you need industrial grade router, switch as well as modem.

I don't disagree that router and modem quality is important, but I don't think this is an issue of that. At the same time, these are ridiculously high ping times. I was pinging before while having this error and was getting ~10 ms to google.ca. Although you don't see any packet loss here, I was experiencing some packet loss to sun.com and download.linino.org. This may be due to current heavy traffic on my network (streaming video).

Note that I am currently testing all this on a 2nd Yun that I have at my office, not the one in the field. The one in the field cannot be communicated with, but I can see it in my hotspot router still.

root@yun01:~# ping sun.com -c 4
PING sun.com (156.151.59.35): 56 data bytes
64 bytes from 156.151.59.35: seq=0 ttl=241 time=275.515 ms
64 bytes from 156.151.59.35: seq=1 ttl=241 time=602.410 ms
64 bytes from 156.151.59.35: seq=2 ttl=241 time=448.691 ms
64 bytes from 156.151.59.35: seq=3 ttl=241 time=394.798 ms

--- sun.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
root@yun01:~# ping download.linino.org -c 4
PING download.linino.org (88.198.63.58): 56 data bytes
64 bytes from 88.198.63.58: seq=0 ttl=50 time=620.248 ms
64 bytes from 88.198.63.58: seq=1 ttl=50 time=515.452 ms
64 bytes from 88.198.63.58: seq=2 ttl=50 time=572.670 ms
64 bytes from 88.198.63.58: seq=3 ttl=50 time=589.146 ms

--- download.linino.org ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss

Does your router support Quality of Service (QoS) ?

QoS is a feature of routers and switches which prioritizes traffic so that more important traffic ( Yun traffic! :P) can pass first. The result is a performance improvement for critical network traffic.

sonnyyu: Does your router support Quality of Service (QoS) ?

QoS is a feature of routers and switches which prioritizes traffic so that more important traffic ( Yun traffic! :P) can pass first. The result is a performance improvement for critical network traffic.

It does, and I appreciate the suggestion, but this is really going off on a tangent. I was having issues using opkg before I was streaming a hockey game :P. This just isn't making much sense to me... I was trying install NRPE by your suggestion, and now I can't really install anything. I never had this problem over the past couple months at all... but that's how it goes. I have restored the Yun to factory and still have the issue.

I have re-run the pings, and they are reasonable even though I am still streaming video. google.ca is back down to ~15 ms. Still unable to use opkg. I do appreciate the quick responses and help trying to get the cause. More recent pings:

root@yun01:~# ping sun.com -c 4
PING sun.com (156.151.59.35): 56 data bytes
64 bytes from 156.151.59.35: seq=0 ttl=241 time=116.917 ms
64 bytes from 156.151.59.35: seq=1 ttl=241 time=120.231 ms
64 bytes from 156.151.59.35: seq=2 ttl=241 time=108.101 ms
64 bytes from 156.151.59.35: seq=3 ttl=241 time=103.129 ms

--- sun.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 103.129/112.094/120.231 ms
root@yun01:~# ping download.linino.org -c 4
PING download.linino.org (88.198.63.58): 56 data bytes
64 bytes from 88.198.63.58: seq=0 ttl=50 time=119.817 ms
64 bytes from 88.198.63.58: seq=1 ttl=50 time=129.606 ms
64 bytes from 88.198.63.58: seq=2 ttl=50 time=121.974 ms
64 bytes from 88.198.63.58: seq=3 ttl=50 time=119.315 ms

--- download.linino.org ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 119.315/122.678/129.606 ms

This is where the shell hangs when trying to get package list:

root@yun01:~# opkg update
Downloading http://download.linino.org/dogstick/all-in-one/latest/packages//Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.
Downloading http://download.linino.org/dogstick/all-in-one/latest/packages//Packages.sig.

I also cannot update the package list in Luci.

I have tried using the Yun's Wifi Adapter rather than Ethernet, but the issue persists.

Is there anything that might not be restored to factory default when I restore the Yun that could be causing this?