Go Down

Topic: Linino Locks up after ~12-24 hours Running php Script every minute (Read 19618 times) previous topic - next topic

sonnyyu

#15
Mar 22, 2014, 10:13 pm Last Edit: Mar 22, 2014, 10:30 pm by sonnyyu Reason: 1
post
Code: [Select]
ping sun.com
ping download.linino.org


reboot/power cycle wifi router, power cycle ethernet switch if you have one, power cycle ISP modem as well. It might be internet problem.

if you are working on 7/24 project then you need industrial grade router, switch as well as modem.



whisprer


post
Code: [Select]
ping sun.com
ping download.linino.org

reboot/power cycle wifi router, power cycle ethernet switch if you have one, power cycle ISP modem as well. It might be internet problem.
if you are working on 7/24 project then you need industrial grade router, switch as well as modem.


I don't disagree that router and modem quality is important, but I don't think this is an issue of that. At the same time, these are ridiculously high ping times. I was pinging before while having this error and was getting ~10 ms to google.ca. Although you don't see any packet loss here, I was experiencing some packet loss to sun.com and download.linino.org. This may be due to current heavy traffic on my network (streaming video).

Note that I am currently testing all this on a 2nd Yun that I have at my office, not the one in the field. The one in the field cannot be communicated with, but I can see it in my hotspot router still.

Code: [Select]

root@yun01:~# ping sun.com -c 4
PING sun.com (156.151.59.35): 56 data bytes
64 bytes from 156.151.59.35: seq=0 ttl=241 time=275.515 ms
64 bytes from 156.151.59.35: seq=1 ttl=241 time=602.410 ms
64 bytes from 156.151.59.35: seq=2 ttl=241 time=448.691 ms
64 bytes from 156.151.59.35: seq=3 ttl=241 time=394.798 ms

--- sun.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss


Code: [Select]
root@yun01:~# ping download.linino.org -c 4
PING download.linino.org (88.198.63.58): 56 data bytes
64 bytes from 88.198.63.58: seq=0 ttl=50 time=620.248 ms
64 bytes from 88.198.63.58: seq=1 ttl=50 time=515.452 ms
64 bytes from 88.198.63.58: seq=2 ttl=50 time=572.670 ms
64 bytes from 88.198.63.58: seq=3 ttl=50 time=589.146 ms

--- download.linino.org ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss

sonnyyu

Does your router support Quality of Service (QoS) ?

QoS is a feature of routers and switches which prioritizes traffic so that more important traffic ( Yun traffic!  :P) can pass first. The result is a performance improvement for critical network traffic.

whisprer


Does your router support Quality of Service (QoS) ?

QoS is a feature of routers and switches which prioritizes traffic so that more important traffic ( Yun traffic!  :P) can pass first. The result is a performance improvement for critical network traffic.


It does, and I appreciate the suggestion, but this is really going off on a tangent. I was having issues using
Code: [Select]
opkg before I was streaming a hockey game :P. This just isn't making much sense to me... I was trying install NRPE by your suggestion, and now I can't really install anything. I never had this problem over the past couple months at all... but that's how it goes. I have restored the Yun to factory and still have the issue.

I have re-run the pings, and they are reasonable even though I am still streaming video. google.ca is back down to ~15 ms. Still unable to use
Code: [Select]
opkg. I do appreciate the quick responses and help trying to get the cause. More recent pings:

Code: [Select]
root@yun01:~# ping sun.com -c 4
PING sun.com (156.151.59.35): 56 data bytes
64 bytes from 156.151.59.35: seq=0 ttl=241 time=116.917 ms
64 bytes from 156.151.59.35: seq=1 ttl=241 time=120.231 ms
64 bytes from 156.151.59.35: seq=2 ttl=241 time=108.101 ms
64 bytes from 156.151.59.35: seq=3 ttl=241 time=103.129 ms

--- sun.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 103.129/112.094/120.231 ms


Code: [Select]
root@yun01:~# ping download.linino.org -c 4
PING download.linino.org (88.198.63.58): 56 data bytes
64 bytes from 88.198.63.58: seq=0 ttl=50 time=119.817 ms
64 bytes from 88.198.63.58: seq=1 ttl=50 time=129.606 ms
64 bytes from 88.198.63.58: seq=2 ttl=50 time=121.974 ms
64 bytes from 88.198.63.58: seq=3 ttl=50 time=119.315 ms

--- download.linino.org ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 119.315/122.678/129.606 ms

whisprer

#19
Mar 23, 2014, 02:41 am Last Edit: Mar 23, 2014, 03:02 am by whisprer Reason: 1
This is where the shell hangs when trying to get package list:

Code: [Select]
root@yun01:~# opkg update
Downloading http://download.linino.org/dogstick/all-in-one/latest/packages//Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.
Downloading http://download.linino.org/dogstick/all-in-one/latest/packages//Packages.sig.


I also cannot update the package list in Luci.

I have tried using the Yun's Wifi Adapter rather than Ethernet, but the issue persists.

Is there anything that might not be restored to factory default when I restore the Yun that could be causing this?

Jantje

I don't think 100ms for a ping is reasonable. Here are mine:
Code: [Select]
64 bytes from 192.168.1.100: icmp_seq=1 ttl=64 time=7.57 ms
64 bytes from 192.168.1.100: icmp_seq=2 ttl=64 time=1.84 ms
64 bytes from 192.168.1.100: icmp_seq=3 ttl=64 time=1.47 ms
64 bytes from 192.168.1.100: icmp_seq=4 ttl=64 time=1.17 ms
64 bytes from 192.168.1.100: icmp_seq=5 ttl=64 time=1.62 ms


Just ran the opkg command on my yun. Works like a charm
Code: [Select]
root@Marvin2:~# opkg update
Downloading http://download.linino.org/dogstick/all-in-one/latest/packages//Packages.gz.
Updated list of available packages in /var/opkg-lists/barrier_breaker.
Downloading http://download.linino.org/dogstick/all-in-one/latest/packages//Packages.sig.
Signature check passed.

Are you sure your firewall is not blocking something? does wget work?
Code: [Select]
root@Marvin2:~# wget http://download.linino.org/dogstick/all-in-one/latest/packa
ges//Packages.sig
Connecting to download.linino.org (88.198.63.58:80)
Packages.sig         100% |*******************************|   543   0:00:00 ETA
root@Marvin2:~# ls
Packages.sig
root@Marvin2:~#
Do not PM me a question unless you are prepared to pay for consultancy.
Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -

whisprer

Sure, 100 ms ping times aren't great, but I am overseas. I tried all this on a different internet connection (4G hotspot from mobile phone) and am once again able to install packages... for now. So it seems it is a router/modem issue, but I didn't change anything and this happened quite randomly. So there is still something wrong, but at least I have isolated it from the Yun... unless it was an Ethernet issue and it just happens to be working because of using Wifi. Now on to implementing NRPE...

The new ping times  :)!:

Code: [Select]
root@yun01:~# ping download.linino.org -c 4
PING download.linino.org (88.198.63.58): 56 data bytes
64 bytes from 88.198.63.58: seq=0 ttl=47 time=453.278 ms
64 bytes from 88.198.63.58: seq=1 ttl=47 time=443.713 ms
64 bytes from 88.198.63.58: seq=2 ttl=47 time=422.705 ms
64 bytes from 88.198.63.58: seq=3 ttl=47 time=425.386 ms

--- download.linino.org ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 422.705/436.270/453.278 ms


Code: [Select]
root@yun01:~# ping sun.com -c 4
PING sun.com (156.151.59.35): 56 data bytes
64 bytes from 156.151.59.35: seq=0 ttl=241 time=349.327 ms
64 bytes from 156.151.59.35: seq=1 ttl=241 time=416.751 ms
64 bytes from 156.151.59.35: seq=2 ttl=241 time=383.680 ms
64 bytes from 156.151.59.35: seq=3 ttl=241 time=490.609 ms

--- sun.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 349.327/410.091/490.609 ms

whisprer

Ok, now that that crazy issue with internet / opkg is out of the way, I have some concerns about NRPE and my initial issue.

It appears that the Linino isn't locking up, but that it may just be losing network connectivity. As I said, I can still see it connected to the router when I lose SSH connectivity and it fails to sync to my remote MySQL database. Resetting Linino if it loses internet connectivity is a potential solution, but it would introduce further issues:

My project only logs data locally if it has the correct time. It does this by comparing the latest entry in the sqlite3 database with the current time, and makes sure the current time is later. There may be times where the 4G router internet goes down, but I don't want to reset the Linino because it would cause it to lose the current time and not be able to restore it until the internet comes back up. I need it to continue logging data locally in the case of an internet failure, and my PHP script automatically syncs the data to my MySQL server when internet connectivity is restored.

Any suggestions on how to determine the cause of losing network connectivity after ~12-24 hours or differentiate between Linino losing network connectivity and my 4G router losing internet? (Note that when this issue occurs my 4G router still has internet access... I can login to it remotely.)

On top of this, are there any suggestions on how to reboot the Linino when I can only see it as a connected device in the router? I have this in the field right now and it would be beneficial to avoid a site visit just to reboot Linino.

Thanks for all the help... this is a strange issue!

sonnyyu

Quote
On top of this, are there any suggestions on how to reboot the Linino when I can only see it as a connected device in the router? I have this in the field right now and it would be beneficial to avoid a site visit just to reboot Linino.


If  your router support:


  • Dynamic DNS (DDNS)

  • SSH



Dynamic DNS (DDNS) is a method of automatically updating a name server in the Domain Name System (DNS), often in real time, with the active DNS configuration of its configured hostnames, addresses or other information.

Secure Shell (SSH) is a cryptographic network protocol for secure data communication, remote command-line login, remote command execution, and other secure network services between two networked computers that connects, via a secure channel over an insecure network, a server and a client .

Best candidates is Openwrt router (same OS as Yun).  Now you could login router then from there ssh to Yun.

sonnyyu

Plan B: load Dynamic DNS (DDNS) at Yun.

Code: [Select]
opkg update
opkg install ddns-scripts


http://wiki.openwrt.org/doc/howto/ddns.client

Then setup port forwarding forward port 22 of Yun.

Plan C: same as Plan B, but put Yun at DMZ zone.


whisprer


Plan B: load Dynamic DNS (DDNS) at Yun.

Code: [Select]
opkg update
opkg install ddns-scripts


http://wiki.openwrt.org/doc/howto/ddns.client

Then setup port forwarding forward port 22 of Yun.

Plan C: same as Plan B, but put Yun at DMZ zone.


Thanks for the suggestions, but I don't think this is what I need. I am able to access the router and Yun (SSH) fine for the first ~12 hours of uptime. I don't think I need DDNS... I have a static IP on my 4G router and I haven't been unable to connect to it remotely over the past ~1 week.

whisprer

#26
Mar 23, 2014, 04:58 pm Last Edit: Mar 23, 2014, 05:30 pm by whisprer Reason: 1

Now you could login router then from there ssh to Yun.


Can you elaborate on this last line? Unfortunately, it is impractical to install OpenWRT or similar on my 4G router (NETGEAR MBR1516) and it is impractical to add a second router to the setup. (FYI, my next implementation will use a 4G USB dongle directly plugged into the Yun.) I have a static WAN IP on the router from my ISP, I have the Yun set to a static IP on the router's subnet, and I have had port 22 forwarded to the Yun from the beginning. As I mentioned above, after a Linino reboot I can SSH to the Yun fine (until ~12-24 hours later) and I can also access the Luci config pages (I also have port 80 forwarded to the Yun).

The problem is that after ~12-24 hours the Yun is essentially unreachable, other than the fact that it shows up here:
After a modem reboot, the device comes back as 'attached'.


sonnyyu

back to original point:


  • use nagios to monitor or profile system metrics.

  • put detect internet connection script at cronjob that runs every ~10 minutes, reboot machine if connection drop.



whisprer


back to original point:


  • use nagios to monitor or profile system metrics.

  • put detect internet connection script at cronjob that runs every ~10 minutes, reboot machine if connection drop.





Yes, this will work, but will have some undesirable consequences -> downtime (minor), but also the device would stop logging even if all is well and the 4G internet actually goes down. Perhaps there is a way to check the connection between the Yun and the router instead? I'd still like to get to the bottom of this issue because I don't think it should be happening at all. I am buying a 3rd Yun to test because my 2nd one isn't exhibiting these issues.

I was able to regain connection to my Yun in the field by changing the static IP in the 4G router setttings. I changed it from 192.168.0.10 to .180 and after some time, it obtained the new IP. I don't think it immediately got the new IP address after rebooting the 4G router. With the new IP of .180 and the ports forwarded to match, I am once again able to SSH and HTTP to the Yun. The Yun shows ~3 days of uptime, so it never restarted, confirming the suspicion that it is just losing internet connectivity.

Any ideas? I am thinking it has to do with DHCP lease time, even though I have it setup as static in the router. The recurring elapsed time to failure of ~12 hours is just too much to be a coincidence. I have already found some issues with the MBR1516 firmware that caused me a lot of grief. There is a bug where a port forward set up as TCP/UDP reverts to UDP when saved. So then trying to connect via TCP doesn't work. I wonder if there is also a bug with their DHCP or if it was a problem to have the Yun set to static within the DHCP range (now changed range to .2 to .100 and set Yun to .180).

Something clearly isn't right!

sonnyyu


Go Up