Webhooks Cache Resets

Development Tools Cloud IoT
1

Hello,

I'm trying to understand how long it takes for a webhook to flush data. I'm seeing duplicate executions for some devices.

In detail,

I have one webhook created at Google Script which is used for multiple Things on the IOT Cloud.

I've noticed that not all variable values are posted by IOT Cloud, only those that change either by "on change" or "interval" as set in the Thing Setup.

However, I have noticed that for some Things (not all) running the same webhook will produce duplicate data posts.

For example, I have a 'message' variable defined as a String which updates "on change". The webhook sends data to GS and the following code snags the correct information that I'm looking for:

var cloudData = JSON.parse(e.postData.contents);
var thingData = cloudData.values;

for( var i=0; i < thingData.length; i++ ) {
    if( thingData[i].name != 'message' ) { continue; }
    var updated_at = thingData[i].updated_at;
    var mString = thingData[i].value;
}

Thank you,

Chris

2

Hello @hermosaranchllc,

Thanks for the reporting.

What do you mean with duplicate data posts?
Do you receive multiple webhook data with the same updated_at for the same property of a thing?

Thanks,

Stefano

3

Hi Stefano,

Yes, I have Things that use a Webhook to post JSON data to my data server (via Google Script) where the 'updated_at' value for a cloud variable is the same but sent twice in a period of about 40 seconds. Ideally, I would like to bypass Google, but I understand why Arduino won't allow direct data posts to individual servers across your user-base... So we use Google as the intermediary as follows...

Arduino IOT Cloud Webhook (Arduino Server) -> Google Script (Google) -> pHp (my server)

The Google Script code (in the previous post above)

var cloudData = JSON.parse(e.postData.contents);
var thingData = cloudData.values;

for( var i=0; i < thingData.length; i++ ) {
    if( thingData[i].name != 'message' ) { continue; }
    var updated_at = thingData[i].updated_at;
    var mString = thingData[i].value;
}

trims the data to name='message' before sending to my server. I do this to create consistency of data transfer across multiple Things which each have variables not common to each other, but all have the variable 'message' common to each other.

For example, on 2025-04-17 Pacific Standard Time, the data from Arduino IOT Cloud Webhook, Thing 'xyz'

thing_id='xyz'
name='message'
updated_at='2025-04-18T03:23:48.474Z'
value='Pump ON'

was sent twice: first at 20:23:51 and then again at 20:24:31 (Pacific Standard Time).

You'll note that my server received the first posting about 3 seconds after the Webhook was triggered. Then again about 40 seconds afterwards. Those two PST timestamps are my server logging the posts from Google.

My hypothesis is that Arduino Cloud has a cache on cloud variables when it packages JSON data in Webhooks for the following reasons:

  1. The Google Script logs show postings from Arduino to Google in 15+ second intervals for the same Thing since the Thing has variables that update in 15+ second intervals where 'message' updates only 'on change'.

  2. I can remove the line of code in Google Script

if( thingData[i].name != 'message' ) { continue; }

to send more data to my pHp server and see that 'message' has the same 'value' but other variables have different values.

  1. Letting the larger JSON packet post to my server in 2) above, the variable 'message' eventually disappears from the JSON packet only coming back when it's value has changed.

The duplicates impact down-stream actions like sending alarms and notices on our side, and also triggering other device actions through the API.

I was able to create a solve on my side (think Swap Memory) by logging each instance of: thing_id, name='message', updated_at, value, myLocalTimeStamp, and then comparing any new instance against prior instances to skip duplicates. I then flush older instances every few days to keep the database small.

This solve is a fast process on my side, doesn't involve a lot of computing overhead right now nor does it impact API or Modbus write commands with a bunch of latency over a few seconds. The 21st Century is truly amazing. So personally, I don't care at this time. And looking forward at customer growth in the near term, I don't think a big computational slow-down is going to occur, nor will our bandwidth on the ISP side be an issue.

But! maybe Arduino can save a little time, overhead and money if you can clear out a caching issue given the number of users you have using Webhooks???

And by the way, if you do manage to fix this, I'll eventually see it since my instance logging will reduce in size.

Finally, I don't think this technically solves this Forum post, so I won't close this posting at this time.

Hope this helps and thank you!

Chris