Go Down

Topic: HttpClient sketch, crashing after about three cycles. <possible solution> (Read 4264 times) previous topic - next topic

PCWorxLA

Quote
Considering that the 1.5.5 IDE was just released in early December (release notes from Nov.28), I am not sure if it is a good idea to replace the bridge.cpp file with one from a post in October or even earlier...


Good point Ralf... first day on the forum and in such a hopeful rush to solve the problem I was not observing dates.  Just now I downloaded the NIGHTLY ... which I am guessing contains both the latest IDE and Bridge.cpp

The yun stalled again at 39 cycles of the above code.
I will have access to my Yun system again this weekend, I will give it a try and see what happens to me...

As for posting code, it would be a good idea if you get used to surrounding it by code "(#") tags, that makes it easier to read and to copy...  ;)

Ralf

GinVan

Quote
As for posting code, it would be a good idea if you get used to surrounding it by code "(#") tags, that makes it easier to read and to copy...  


Thanks Ralf for the tip... I edited my posts with code tags. (Good features on this forum - lots of helpful systems to learn about).

Also... In my eagerness (and apparent frustration - and lack of sleep while trying to resolve the http client issue :) )  I posted a rather wordy response on another thread that seemed to be related.  The changes to the files mentioned as a solution there (though I am not 100% clear if they are actually specifically related to this issue) did not solve the http client issue:

http://forum.arduino.cc/index.php?topic=196091


Federico Fissore

Yes, the nightly contains all the fixes available for the IDE. However, and just for the record, it does not fix or update your yun.

Your sketch seems right to me.
When you say "crashes" or "hangs", what are you experiencing exactly? What do you expect and what does not happen?
Have you upgraded the Yún? If you've just got it, then it needs to be upgraded! Check out the tutorial

GinVan

Hi Frederico... thanks for the reply and for also trying to help.
In my previous posts I provided two code examples.  The first version has more comments in the code with details regarding how the code seems to hang.
My code is printing serial output: a count, a marker before client.get and a marker "Fetching posts" showing that the code is has moved past the client get "Posts received".  I am updating page "posts.php" with content dynamically - either with blank content or two integers separated by a space such as "1 20" (1[space]20).  I have noticed that a trailing zero "0" is also received which I assume is the end of file marker "null".
I am running the code with a delay(10000) between client.get fetches.

The serial monitor displays output such as:
69 Fetching posts | Posts received
70 Fetching posts | Posts received  
71 Fetching posts

The display after a number of cycles will not move past "Fetching posts" such as the example above stalled at 71.

I am expecting the code to run continuously without fail so that I can receive and parse content when it is provided.

Code: [Select]
Serial.print(count);
Serial.print(" Fetching posts ");
client.get("http://..../posts.php");
Serial.println("| Posts received");



Also is there something I need to do to update code within my yun?  I edited some files that you mentioned on a previous thread.  There is a link above in my previous post.  Please advise me on how I can get this yun to operate... receiving content from the web continuously, without fail.

Thanks,

G


VideoLiam

Hi GinVan,

I was reading through your posts and just thought of one experiment that might be worth trying.

Have you tried to run your Yun on a different network?

Internal switches have been known to be poor quality, and if you are going via your service providers internet you have no idea what they are using.. Poor switchers can cause all sorts of problems. It is rare, but could be a simple experiment to try just to be sure.

Liam

GinVan

Hi Liam... I would also be curious about the response on another network, and perhaps another host, as well, however, I do not have those options immediately available.  Yet that would seem to me, at this time, to not be the true issue or solution anyway.  It would seem to me (though I am not an expert in the field) that the hypertext transfer protocol overcomes various issues that are anticipated in the transfer of data.   Surely a device such as the yun, that implements it, should run continuously and not hang 26 to 126 cycles into the loop.  When it does hang I have confirmd that the host (and ISP)  are still able to send/receive by refreshing another browser.
If Internet Explorer and Chrome are able to continuously access without fail (perhaps even running under a script that accomplishes the same purpose) than the issues is not with the ISP or host but with the mechanisms of the yun.
Also... It appears, in this thread (and others), that I am not the only one experiencing this problem with the yun.  

Further to those points I have these considerations that I will be looking for solutions for:
Lets say that the yun is running applications we consider to be important (maybe not a life sustaining human medical device :) ) but lets use an example we have seen before - feeding your pet or watering your plants, or...( let your creative imaginations take over)...
What if the yun crashes, what if the ISP or Host goes down for an hour?  Is there software/hardware solutions within the yun that we can implement to recover?  Is restart reliable enough?

The goal, again: host and ISP reliability assumed, is to use the yun to receive content reliably and indefinitely.

The other important question - what if we transfer the yun, running it's application, to another network that we as developers do not have access to... And without giving the end user access to the yun's control panel, how could a simple network/user/password interface be implemented to allow the user to connect to their own network?

G

Federico Fissore

Wow lots of input, I'll try to give answers...
Regarding:
- other users having similar problems: the biggest issue with the yun is that most of its power comes from the easiness in accessing a network, but unfortunately lots of things can break during a network transmission. I believe most of the problems arose so far are network related corner cases, hence they are hard to debug. I had no way to reproduce most of them (and the other became issues and some already got fixed)

- what happens if your ISP fails: it happens just what happens with your pc, you start saying "google is not working, the internet is broken!" :) Restarting the yun, in this case, is not a solution as it's not a solution with your pc

- how to give our configured/flashed yun to someone else, installing it on a different network. The simplest solution is you configuring it or you giving instructions for configuring it through the webpanel. You can develop your own webpanel, or provide an interactive sketch to configure network, but it's more work.

- finally your 71-then-hanged issue: as said many things can break during a network connection. I would try to debug the "finger pointing" way.

Let's point the finger to your sketch: does running the HttpClient example show the same behaviour? Maybe you want to modify it slightly so that it prints a counter as well.
Let's point a finger to yun's wifi: can you reset wifi conf (so that is restarts in access point mode), plug in an ethernet cable and rerun either your sketch or the HttpClient example?
Let's point a finger to the bridge: can you try running the following shell script from yun linux prompt?
Code: [Select]

# with these number this will run for about 27 hours
for n in `seq 1 10000`
do
curl http://arduino.cc/asciilogo.txt
sleep 10
done

Let's point a finger to your network as VideoLiam suggested: can you connect to another network and retry? (maybe you're at home, then connecting it to your office or school or friend's network will do)

I have no other targets for my finger right now. Let's see what comes out of these tests.
Have you upgraded the Yún? If you've just got it, then it needs to be upgraded! Check out the tutorial

gkzsolt

Ginvan,

I run about your post today and beeing impressed about how long it has been discussed I made some experiments myself. First I reproduced the "gibberish in HttpClient sketch after 4-5 times" (I have IDE v. 1.5.4 installed, with the buggy Bridge). Then I applied the patch given by Federico in some earlier post ("avr: Fixed timeout in Bridge::transfer()" in https://github.com/arduino/Arduino/commit/ca90bf75cb1e256f728e879dc83d17bf3e02197b) (so I opened Bridge.cpp and modified 5 to 10 in the 2 lines given), because I have more confidence in this than to replace the file (which have been worked as well, I guess). AFter this I restarted the HttpClient sketch, which works for ~2 hours now without problems. If one cycle takes 7 seconds (5 + the time to get the request), it is about 9/min, so it run for at least 1000 times without problem.

So I would look after the network stability in your case, the Arduino seems to work and it is reliable enough for me.

Cheers,
   Zsolt

MattS-UK


It would seem to me (though I am not an expert in the field) that the hypertext transfer protocol overcomes various issues that are anticipated in the transfer of data.


Hello GinVan
I would describe myself as a network engineer who sometimes plays with software development.  I see things from a different perspective to software developers who sometimes play with networks.

Your assumption, that HTTP overcomes transmission issues, is quite wrong.  HTTP is an application protocol (layer 7).  Application protocols depend on the protocol stack beneath, (layers 1 to 6) and presume they are functioning correctly.  For instance, a broken cable (layer 1) will break HTTP, and every other protocol above Layer 1.  A poorly configured subnet (layer 3) will break HTTP, and every other application layer protocol.

Quote
Surely a device such as the yun, that implements it, should run continuously and not hang 26 to 126 cycles into the loop.  When it does hang I have confirmd that the host (and ISP)  are still able to send/receive by refreshing another browser.


Software developers new to network programming, have a habit of neglecting the dynamic nature of networks.  The things they tend to forget are;
+ Network access has a cost, measured in CPU and RAM.
+ What asynchronous means.
+ Transient fault conditions are a constant.
+ A protocol defines the means of cooperation but does not provide a method of implementation.
+ One observable symptom may describe many different fault conditions.

Put together, these attributes present your application with a number of failure modes (edge cases) which will cause an application to fail eventually, unless the application provides the functionality to handle  each failure mode gracefully.

Quote
If Internet Explorer and Chrome are able to continuously access without fail (perhaps even running under a script that accomplishes the same purpose) than the issues is not with the ISP or host but with the mechanisms of the yun.


No. No and again no.  The developers of Explorer and Chrome are experienced network programmers.  They have learnt the lessons I mentioned and have adapted their code to handle the edge cases gracefully.

Quote
Also... It appears, in this thread (and others), that I am not the only one experiencing this problem with the yun.


Statements like that cause me a great deal of frustration in my day job ;)  One observable symptom may describe many different fault conditions.  You are observing your problems.  Other people are observing other problems.  They may or may not be the same problems.  Try to focus on your own problems and no one else's.  

I have commented your code to highlight the things I would do differently.  I do not know whether these are your problems, only that such coding techniques are often problematic in network environments.

Code: [Select]

void loop(void)  {
 httpClient client;
 client.get("http://..../posts.php")  // a blocking call with an external dependancy
 
 while(client.available()) {
    //while the input buffer contains some_number of bytes
    command = client.parseInt();
   duration = client.parseInt();
   zero = client.parseInt();
  //read a finite number of bytes (6) from the buffer
 }
 client.get("http://.../clearposts.php");  //another blocking call
 delay(10000);  //another blocking call.
}


Here is a more graceful version of your while loop.
Code: [Select]

uint16_t bytesAvailable = client.available();
if (! (bytesAvailable < 6) ) {
 for (uint16_t i = 0; i < 6; i++) {
   command = client.parseInt();
   duration = client.parseInt();
   zero = client.parseInt();
 }
}


You should be able to remove the blocking behaviour of the .get(url) method by using the httpClient.getAsynchronously(url) method, although it appears the documentation has a glaring typo.
http://arduino.cc/en/Reference/YunHttpClientGetAsynchronously

The blocking caused by the delay(10000) can be avoided using a global variable and millis()
Code: [Select]

uint32_t periodStart = 0;
void loop(void) {
 uint32_t period = millis() - periodStart;
 if (period > 10000) {
   //do stuff
 }
}


Quote
Is there software/hardware solutions within the yun that we can implement to recover?  Is restart reliable enough?


I started out using the WiFi shield and can tell you for certain, that device will not provide reliable TCP performance unless the firmware is re-written.  The Yun is a leading edge device, providing a great deal of potential but it is early days.  The possibilities and the edge cases, are still being identified.

Quote
The goal, again: host and ISP reliability assumed, is to use the yun to receive content reliably and indefinitely.


Reliable is achievable.  Indefinitely is not achievable.  I have (proper) servers in the wild with uptimes measured in years but they will eventually need to be rebooted.  The applications being served, rarely make it past a month or so without a restart being required.

Quote
And without giving the end user access to the yun's control panel, how could a simple network/user/password interface be implemented to allow the user to connect to their own network?


As I often have to explain to my customers, inserting words like 'just' or 'simple' into a sentence, does not change the complexity of what the sentence describes.  Until such time IPV6 becomes ubiquitous, we are stuck having to write our own IPV4 subnet provisioning scripts.  Without remote access, we are stuck with the old fashioned support methods; documentation no one wants to read and trying to decipher the words, it does not work.

Best of luck and I hope the practical suggestions prove useful.


Federico Fissore


Here is a more graceful version of your while loop.
Code: [Select]

uint16_t bytesAvailable = client.available();
if (! (bytesAvailable < 6) ) {
 for (uint16_t i = 0; i < 6; i++) {
   command = client.parseInt();
   duration = client.parseInt();
   zero = client.parseInt();
 }
}



Thank you Matt, I'll share this piece of wisdom in future posts

Quote

As I often have to explain to my customers, inserting words like 'just' or 'simple' into a sentence, does not change the complexity of what the sentence describes.


Again thank you: I foresee many chances of using this phrase.
Have you upgraded the Yún? If you've just got it, then it needs to be upgraded! Check out the tutorial

GinVan

Thanks to each of you who has provided input.
The new network did not prove to work any better.
I will be testing the other suggestions and code provided and then comment asap.

Thanks,

G

GinVan

While using the code sample from MattS-UK... I get this error in the arduino IDE:

Quote
Exception in thread "EventThread COM11" java.lang.NullPointerException
   at processing.app.Serial.serialEvent(Serial.java:176)
   at jssc.SerialPort$EventThread.run(SerialPort.java:1096)


The code I was testing ran longer than other codes... this failing at 197 cycles:

Code: [Select]
uint16_t bytesAvailable = client.available();
if (! (bytesAvailable < 6) ) {
  for (uint16_t i = 0; i < 6; i++) {
    command = client.parseInt();
    duration = client.parseInt();
    zero = client.parseInt();
  }
}

Go Up