Possible problem with multiple calls to client.getsasyncronous after several day

Has anyone else experienced a problem with the linino side of the bridge library slowing down after a few thousand calls to getAsynchronous URL.

This morning I found my project has slowed updates to thingspeak down to a crawl. My project is remote so I have a remote soft reset in it using the quick and dirty
asm volatile (" jmp 0"); Everything gets properly reinitialised in the setup routine so I know this works nicely after months of rock solid testing.

Once every 11 seconds a function sends the latest readings to the thingspeak server. Thingspeak also allows me to retrieve any waiting talkback commands at the same time hence I send the url using get async and the waiting commands are returned into client. available.

void UpdateThingSpeak() 
{
        if (millis() - LastUpdateTime > 11000)
        {
                URL[0]='\0';                                                                                            
                                                                                                                         
                strcat(URL,"http://");                                                                                 
    		                              //code to build the rest of the url goes here etc etc
                strcat(URL,'\0');
                
                while(client.available() > 0) // Before we start, if anything is available in the client buffer
                {
                        client.read();        // We call a read to clear out any rubbish in the buffer
                }
                client.getAsynchronously(URL);// Tell Linino to 'GET' the url in the background which will return the command string                
                LastUpdateTime = millis();    // Reset the timer
        }
}

Then once a second my main loop checks to see if there were any commands

void CheckForCommands()  
{
        char inChar='\0';
        byte i=0;                                                                                                       
        byte c=0;                                                                                                       
        RemoteCommandString[0]='\0';                                                                                    
        while (client.available())                                                                                      
        {
                inChar = client.read();  
		// code to to stuff with it goes here                                                                               
	}
}

This morning I noticed the slowdown. not there last night and there this morning. The controller hasnt been hard reset for a good few days. Probably 3 and half days or so. Around 26000 calls to get async

The slowdown was a right pain. Rather than getting updates every 11 seconds they seemed to be coming in every 3 minutes or so. I eliminated everything else it could be. server problems. local and remote internet problems. Because the remote jump is triggered by a talkback command and I can see when they are being executed, I noted that the problem seemed to be the getasync was not firing every 11 seconds as usual. Eventually a command was executed and a asm volatile reset occured but the problem didnt go away. At this point it started to look less likely it was an arduino side issue.

My project is in australia and I’m in the UK. I got my freind in Aus to do a reset on the 32u4 side only. It came back up but the problem persisted. Hence Im fairly sure now that the problem doesnt lie in the compiled bridge library on the 32u4

finally. I asked him to unplug the yun and plug it back in. Bingo. the problem went away.

Trouble is. Im not sure if its a one time problem or something which will reoccur. Im constantly testing this and revising things so it doesnt often get left for a few days. but ultimately the final project will.

So I am wondering if constantly calling getAsync is leaking memory or suchlike on the linino side. I seem to constantly call a getasync but theres no advice on if I should be telling the linino yes thats fine. I’ve got what I needed you can clean up that request.

To be fair I find the examples for getasync on the arduino site far from clear for a hobbyiest product… I fiddled until I got it to work but I’m not sure if what i’ve done is best practice. all i know is it worked for a long time but this is probly the first time i’ve not fiddled with it and just let it get on with the task for a few days.

Perhaps a one for federico?

mearsy25: Has anyone else experienced a problem with the linino side of the bridge library slowing down after a few thousand calls to getAsynchronous URL.

::::SNIP::::

mearsy25, After along diatribe, which i deleted, I noticed you are using a synchronous and Asynchronous call. You need to use one or the other, not both.

Jesse

Jesse,

Thanks for the update. Putting aside for one moment why this code seems to have run ok and now isn’t. And eliminating the possibility its a network connection issue or the yun has gone bad. (Yes lets eliminate that from the thought experiment)

Please would you mind expanding on how you would modify those two bits of code to be both async as frankly the documentation is, was and has always been totally crap on what is a hobbiest product.

client.getasync is definately a call to an async process so my question is how would you modify my other client.read or client.available to be appropriate for the task. A quick mod of the code below would help clear things up

P.S Heres an attachment of 1600 record. The issue with it not sending update starts at record 1073 at 12:54:02 and begins to get gradually worse from there. In case the chart doesn’t clearly show it the text document will

Usual caveats. I can’t post all the code. Proprietary algorithms. Its worked. now it doesn’t but just in case I’m looking for best practice on how to properly read a get async in the “async” manner as you suggested

Thanks

Chris

logs.txt (139 KB)

mearsy25: Once every 11 seconds a function sends the latest readings to the thingspeak server.

Probably not related, but just in case: ThingSpeak limits the data update rate to once every 15 seconds. My experience is that a second update within 15 seconds simply returns an error. But is it possible that they are doing other rate limiting after a long enough continual update session? Probably not the issue here, I'm just brainstorming possible scenarios.

Unfortunately, I don't have other help to offer: my experience with ThingSpeak is using a very different development environment -- I haven't tried updating it using any kind of Arduino.

Thank you shapeshifter.

Yes, your quite correct. normally and update within 15 seconds would result in a zero rather than an entry id. However I discovered that there is an exception to the rate limit. if you are using talkback and doing an update and execute of a channel at the same time you can update more frequently. I chose every 12 seconds for 5 updates a minute.

interestingly the as side one seemed to give up updating after 1708 records.

So it starts fine. then begins to slow down updates then eventually the situation gets so bad it just stops doing it.

this is new behaviour. Ive been running this code for quite some time and never seen this. today. I found that the problem seems to occur reasonable quickly. within 1200 records. its never done this before. if it was flaky wifi i’d expect it to be more sporadic. I wonder if something on the lining side is goosed.

I am wondering if its the yun i aus which has developed a problem. possibly flaky wifi.

I have a second test yun here in the uk running the same code. its brand new out of the box. it looks like a remote issue.

Im still interested to know how I am using a mix of sync and async methods as alluded to above?

mearsy25:
Jesse,

Thanks for the update. Putting aside for one moment why this code seems to have run ok and now isn’t. And eliminating the possibility its a network connection issue or the yun has gone bad. (Yes lets eliminate that from the thought experiment)

Please would you mind expanding on how you would modify those two bits of code to be both async as frankly the documentation is, was and has always been totally crap on what is a hobbiest product.

::::SNIP::::

Usual caveats. I can’t post all the code. Proprietary algorithms. Its worked. now it doesn’t but just in case I’m looking for best practice on how to properly read a get async in the “async” manner as you suggested

Thanks

Chris

Chris,
I going to forego making a joke on your suggestion of “expanding on how you would modify those two bits of code”. Seriously, you can’t claim some “Proprietary algorithms” has any sort of value and not understand the difference between synchronous and Asynchronous.

So, in an effort to help Google: synchronous vs Asynchronous

I’m being a bit arrogant because I write “Proprietary code” all the time. Your suggestion is JUST … And I’ll stop there.

Here is another link: http://stackoverflow.com/questions/748175/asynchronous-vs-synchronous-execution-what-does-it-really-mean

Jesse

Jesse,

I’m perfectly well aware of the difference between syncronous and asyncronous. Just as I hold an strong interest in the mechanisms underpinning the hyperphosphorylation of tau protein aggregates. You see Jessie what you appear to fail to understand is that not everyone posting on here is in your intellectual shadow. Each has their own strength. There may be many things which you know and I do not but the opposite is also true, so please. Don’t go treating everyone as an idiot. It only reflects badly on you.

I will restate again. What is not clear from the documentation is how having called the client.getAsynchronously(URL) I am supposed to correctly read that result in an asyncronous manner as per your suggestion below.

“I noticed you are using a synchronous and Asynchronous call. You need to use one or the other, not both.”

It strikes me that if I am constantly requesting the underlying linux side to get something asynronously but never telling it thanks. I’m got what I needed. I could be ending up with a load of calls stacked up on the linux side which might be causing an issue. It’s simply an honest request for input from people who know more about that side of things. It doesnt warrant your tone.

Whilst I am in no way convinced that this is the root cause of the problem and suspect something has gone bad on the linino or connection side. I’d rather code it in the correct manner for the sake of completeness.

Your being very hostile. This forum is a place for people to come and learn from others experience. There are plenty of other places on the internet you can vent and rant but a forum for people who enjoy learning to code as a hobby is not one.

Even if my code didn’t contain stuff which is considered proprietary. I wouldn’t wish to share the whole code because.

a.) Its irrelevant to the problem
b.) People like you are too easily distracted and would rather tear into all the other things you perceive I am doing wrong in my code rather than focussing on the issue in question.

Seeing as you set your stall out with the above comment that I am using and incorrect mix of calls perhaps you’d like the opportunity to validate your superiority and furnish us all with the correct “asyncronous” manner of reading a call made using client.getasyncronous(url).

That way, you’ll have redeemed yourself by contributing something meaninful to the discussion and someone in future may find your post of assistance.

I would once again like to thank Shapeshifter for his polite and concise suggestions.

For future reference If anyone would actually like to listen to a load of arrogant uniformed illogical drivel , might I suggest they head over to the guardian’s web site to read what thist this bloke has to say.

Now either play nice and be a helpful member of this forum or be on your way. Theres a good chap.

680394-russell-brand.jpg

Hello mearsy25,

Maybe the problem is due to a non-correct management of the curl process handle inside the HttpClient class.

This may cause, after a lot of call of the getAsync method, some memory leak.

To solve your problem try to do two things:

1) Always check if the get is still running before starting a new one with client.running(). In this way you avoid get calls stacking. You use a 11 seconds interval but maybe the previous call is still active because of some network problem.

2) After you read the response of the getAsync or you know that it finished its work, call the client.close() method in order to close the handle of the last call. This has to be done before you call again the getAsync or the old handle will be overwritten. ( you can close it between the client.running call and the getAsync one)

If you use the normal get, always use the close method after it.

The httpclient class derives from the process one, so it needs to be closed after its operations.

All I said to you is just an experiment, I tried to figure out the problem reading your description and looking at the Bridge code.

Tell me if that works

Angelo

Thank you Angelo. Now thats an example of a helpful friendly response. I will review the code accordingly.

For the time being I've logged into the remote yun. I can see the problem seems to be poor wifi signal. 32%

It drops out constantly for periods of time. I can run a ping -t from the local laptop and watch it get several pings with large latency then a long run of request timed out's then back to normal again.

It seems my guy in Aus has built a brick plant enclosure and put the yun inside a steel case within the brick which is causing the issue.

For the sake of completeness When I log into the Luci console on my local test rig I can see perhaps 10 open connections to thingspeak.com. The figure always seems to stay at around 10 so I figure old connections are dropped as they time out.

Its not causing a problem but your suggestion of how to correctly read the data is brilliant.

Thanks a million

mearsy25: For the sake of completeness When I log into the Luci console on my local test rig I can see perhaps 10 open connections to thingspeak.com. The figure always seems to stay at around 10 so I figure old connections are dropped as they time out.

Yes, ThingSpeak tries to keep the connection open (it includes a keep-alive header in its response.) The idea is that the remote node (your Yun in this case) can then re-use the connection for the next request and save some network overhead. The connection does time out eventually, but it probably is a good idea to go ahead and close it as Angelo instructs.

And before you ask, no, I don't know how to re-use the active connection on the Yun. ;)

The HttpClient just uses the process class to run a curl command. You can, if you want, directly use the Process+curl combination to have the same effect.

I think re-using open connections is a curl-related option, but like ShapeShifter, I don't know how to do that :)

Angelo9999: ::::SNIP::::

I think re-using open connections is a curl-related option, but like ShapeShifter, I don't know how to do that :)

It can be complex.

Tutorial: Using cURL to automate HTTP jobs http://curl.haxx.se/docs/httpscripting.html