Sketch to http post to thingspeak and what I have learned so far

zbelding · June 3, 2020, 3:34am

This is what I have learned so far from my work with the Arduino MKR NB 1500:

Signal

I am in a low signal area. The SARA R4 seems to be very sensitive to low signal. It will get stuck sometimes during certain conditions. The only way to recover is a hard reset. My sketch will hard reset after 20 response timeouts in a row.

More to come later, time to sleep...

The sketch will update 3 thingspeak fields every 10 mins. It will send out the battery voltage, a random number, and the number of modem resets. If you have a thingspeak account, try it out.

zbelding · June 3, 2020, 3:51am

Here is a link:

zbelding · June 3, 2020, 7:52pm

continued from first post...

TCP Socket Commands

TCP socket open/close commands(AT+USOCO & AT+USOCL) can take up to 2 mins depending on signal strength. Sometimes after waiting, the modem will respond ERROR. If this happens, it must be tried again. I have found that an ERROR response is more likely for a TLS socket. I would assume this is because of the extra overhead of the TLS connection. The socket close command(AT+USOCL) can be configured with a "Asynchronous close flag". This means the modem will immediately respond with OK, but it will close the socket on its own in the background. This is helpful if you have any errors TCP connection and you want to try again immediately without waiting 2 mins for the socket to close.

I believe these long wait times with TCP sockets could explain some of the issues people are having with MQTT and http. These are both based on TCP sockets. Many times these socket commands reply quickly, but in the real world where signal can vary greatly, they can take a lot longer.

In my sketch, I am opening a TCP socket to api.thingspeak.com, then sending some data in http post format. After opening the socket and sending the data, I wait for a URC from the modem telling me the response is ready to be read. Sometimes, this URC is not received, even though I know the data was sent. In this case I start the process over again. When a response does come in, I check it to make sure it contains response code: "200 OK". This is confirmation that the data was successfully sent. I am not sure why the response gets lost sometimes...

icole · June 3, 2020, 8:07pm

I have found that I can power the module from a USB 3.0 port on my PC or from a powered USB hub. I believe that a lot of the MQTT problems are with the long time for the response. I've set the timeout for a send to MQTT to 4 minutes which results in things working somewhat better (usually takes longer to hang). When the timeout occurs on a send to MQTT broker generally nothing good happens after that. In m case my code never gets control again thus can't reset and clean up. I'm getting frustrated and now looking for another solution even though I've bought 2 MKR1500 boards. I appreciate you sharing your work.

zbelding · June 4, 2020, 6:17pm

Here is the last 8 hrs. You can see 4 automatic modem resets during this time.

zbelding · June 12, 2020, 12:50am

I have made the following changes:

Fixed a bug in the area of the program that checks modem registration before opening socket.
Changed some function and variable names.

zbelding · June 19, 2020, 11:53pm

I have switched to a directional(yagi) antenna with positive results so far. I was able to aim the antenna with a program that repeatedly sends AT+CESQ to the modem. Pointing the antenna slightly upward seemed to help clear some tree cover...

Socket commands seems much more responsive so far:

18:56:37.512 -> 
19:06:34.122 -> ATE0
19:06:34.224 -> 
19:06:34.224 -> OK
19:06:34.224 -> 
19:06:34.224 -> AT+CEREG?
19:06:34.328 -> 
19:06:34.328 -> +CEREG: 0,1
19:06:34.328 -> 
19:06:34.328 -> OK
19:06:34.328 -> 
19:06:34.328 -> Registered
19:06:34.328 -> AT+CGATT?
19:06:34.431 -> 
19:06:34.431 -> +CGATT: 1
19:06:34.431 -> 
19:06:34.431 -> OK
19:06:34.431 -> 
19:06:34.431 -> Attached
19:06:34.431 -> AT+CGACT?
19:06:34.534 -> 
19:06:34.534 -> +CGACT: 1,1
19:06:34.534 -> 
19:06:34.534 -> OK
19:06:34.534 -> 
19:06:34.534 -> Context Activated
19:06:34.534 -> AT+USOCR=6
19:06:34.670 -> 
19:06:34.670 -> +USOCR: 1
19:06:34.670 -> 
19:06:34.670 -> OK
19:06:34.670 -> 
19:06:34.670 -> AT+USOSEC=1,0
19:06:34.774 -> 
19:06:34.774 -> OK
19:06:34.774 -> 
19:06:34.774 -> AT+USOCO=1,"api.thingspeak.com",80
19:06:35.593 -> 
19:06:35.593 -> OK
19:06:35.593 -> 
19:06:35.593 -> AT+USOWR=1,186
19:06:35.695 -> 
19:06:35.695 -> @
19:06:35.695 -> POST /update HTTP/1.1
19:06:35.695 -> THINGSPEAKAPIKEY: (REMOVED)
19:06:35.695 -> Host: api.thingspeak.com
19:06:35.695 -> Content-Type: application/x-www-form-urlencoded
19:06:35.695 -> Content-Length: 30
19:06:35.695 -> 
19:06:35.695 -> field1=4.117&field2=4&field3=0
19:06:35.832 -> 
19:06:35.832 -> +USOWR: 1,186
19:06:35.832 -> 
19:06:35.832 -> OK
19:06:35.832 -> 
19:06:36.244 -> AT+USORD=1,1024
19:06:36.346 -> 
19:06:36.346 -> 
19:06:36.346 -> +USORD: 1,690,"HTTP/1.1 200 OK
19:06:36.346 -> Date: Fri, 19 Jun 2020 23:06:36 GMT
19:06:36.346 -> Content-Type: text/plain; charset=utf-8
19:06:36.346 -> Content-Length: 5
19:06:36.346 -> Connection: keep-alive
19:06:36.346 -> Status: 200 OK
19:06:36.346 -> X-Frame-Options: SAMEORIGIN
19:06:36.346 -> Access-Control-Allow-Origin: *
19:06:36.346 -> Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
19:06:36.346 -> Access-Control-Allow-Headers: origin, content-type, X-Requested-With
19:06:36.346 -> Access-Control-Max-Age: 1800
19:06:36.346 -> ETag: W/"5f4b75634381196faf857ccc46a291a9"
19:06:36.346 -> Cache-Control: max-age=0, private, must-revalidate
19:06:36.346 -> Set-Cookie: request_method=POST; path=/
19:06:36.346 -> X-Request-Id: 41840dda-ae4d-4dbf-bf7a-47129194163f
19:06:36.346 -> X-Runtime: 0.029371
19:06:36.346 -> X-Powered-By: Phusion Passenger 4.0.57
19:06:36.346 -> Server: nginx/1.9.3 + Phusion Passenger 4.0.57
19:06:36.346 -> 
19:06:36.346 -> 19719"
19:06:36.346 -> 
19:06:36.346 -> OK
19:06:36.346 -> 
19:06:36.346 -> AT+USOCL=1,1
19:06:36.448 -> 
19:06:36.448 -> OK
19:06:36.448 -> 
19:09:28.932 -> ATE0
19:09:29.617 -> 
19:09:29.617 -> OK
19:09:29.617 -> 
19:09:29.617 -> AT+CEREG?
19:09:29.754 -> 
19:09:29.754 -> +CEREG: 0,1
19:09:29.754 -> 
19:09:29.754 -> OK
19:09:29.754 -> 
19:09:29.754 -> Registered
19:09:29.754 -> AT+CGATT?
19:09:29.857 -> 
19:09:29.857 -> +CGATT: 1
19:09:29.857 -> 
19:09:29.857 -> OK
19:09:29.857 -> 
19:09:29.857 -> Attached
19:09:29.857 -> AT+CGACT?
19:09:29.960 -> 
19:09:29.960 -> +CGACT: 1,1
19:09:29.960 -> 
19:09:29.960 -> OK
19:09:29.960 -> 
19:09:29.960 -> Context Activated
19:09:29.960 -> AT+USOCR=6
19:09:30.064 -> 
19:09:30.064 -> +USOCR: 1
19:09:30.064 -> 
19:09:30.064 -> OK
19:09:30.064 -> 
19:09:30.064 -> AT+USOSEC=1,1
19:09:30.199 -> 
19:09:30.199 -> OK
19:09:30.199 -> 
19:09:30.199 -> AT+USOCO=1,"platform.clickatell.com",443
19:09:32.974 -> 
19:09:32.974 -> OK
19:09:32.974 -> 
19:09:32.974 -> AT+USOWR=1,214
19:09:33.077 -> 
19:09:33.077 -> @
19:09:33.077 -> POST /messages HTTP/1.1
19:09:33.077 -> Authorization: (REMOVED)
19:09:33.077 -> Host: platform.clickatell.com
19:09:33.077 -> Content-Type: application/x-www-form-urlencoded
19:09:33.077 -> Content-Length: 46
19:09:33.077 -> 
19:09:33.077 -> content=Smoke!&to=(REMOVED)&from=(REMOVED)
19:09:33.181 -> 
19:09:33.181 -> +USOWR: 1,214
19:09:33.181 -> 
19:09:33.181 -> OK
19:09:33.181 -> 
19:09:33.559 -> AT+USORD=1,1024
19:09:33.663 -> 
19:09:33.663 -> 
19:09:33.663 -> +USORD: 1,367,"HTTP/1.1 200 OK
19:09:33.663 -> Server: nginx
19:09:33.663 -> Date: Fri, 19 Jun 2020 23:09:33 GMT
19:09:33.663 -> Content-Type: application/json
19:09:33.663 -> Content-Length: 155
19:09:33.696 -> Connection: keep-alive
19:09:33.696 -> Strict-Transport-Security: max-age=31536000; includeSubDomains
19:09:33.696 -> 
19:09:33.696 -> {"messages":[{"apiMessageId":"10be3bf1c2a14e3da7e0c8367e8f3e33","accepted":true,"to":"(REMOVED)","errorCode":null,"error":null,"errorDescription":null}]}"
19:09:33.696 -> 
19:09:33.696 -> OK
19:09:33.696 -> 
19:09:33.696 -> AT+USOCL=1,1
19:09:33.797 -> 
19:09:33.797 -> OK

As you can see, It only took a couple seconds to open a socket to thingspeak. Also about the same amount of time to open a SSL socket to clickatell in order to send an SMS.

zbelding · June 20, 2020, 8:14pm

No modem resets required in the last 20 hrs:

scouter9000 · June 21, 2020, 2:14am

I've found very similar results wrt to signal strength, and swapping the antenna. In low signal situations my system fails randomly between both my cellular connection and MQTT connection functions, as opposed to consistently in either one.

My system retries and hard resets the modem after 5 attempts, eventually the watchdog timer resets the whole board. Here is a visualization of my results over 12 hours:

In the high signal strength location my system has been running for over 10 days now with only a couple of watchdog resets, which may be due to the poor modem power handling in nbAccess.begin.

Thanks again for your posts zbelding, they have been invaluable to my attempts to get a viable system online. Just a note that I am unable to view your linked images in the vast majority of your posts.

zbelding · June 21, 2020, 10:17am

scouter9000:
Thanks again for your posts zbelding, they have been invaluable to my attempts to get a viable system online. Just a note that I am unable to view your linked images in the vast majority of your posts.

Thanks for your findings scouter9000. I am about 7 miles away from the nearest AT&T tower (which is the network I am using). This is a hilly, rural area. Before the trees leaved out this spring, I had pretty good luck with the the first antenna I was using. Now with the leaves, the directional antenna is required.

Can you see the pics in this thread now? I changed the links.

zbelding · June 23, 2020, 2:22pm

Only one automatic modem reset required over the past four days since I switched to the directional antenna:

zbelding · June 29, 2020, 10:48am

I have made the following changes:

I changed the <PDP_type> parameter of AT+CGDCONT from "IP" to "IPV4V6". This should be correct for most carriers, but could vary...
I added functionality to handle a modem issue as described below:

Symptoms:

-Modem is responsive
-Modem has plenty of signal
-Modem is registered, attached, and has an IP address
-Modem will not open a socket, no matter how many tries

This is a rare problem, but I have seen it half a dozen times now.

Cure:

-Soft reset of the modem with AT+CFUN=15

I have added a modemSoftResetPending bool that will soft reset the modem in the modemSetup() function. If the rawHttpPost function fails to open a socket 10 times in a row, modemSetup() will be run again with modemSoftResetPending = true.

zbelding · July 13, 2020, 4:39pm

I have fixed: bool chargerConfigFlag ----> volatile bool chargerConfigFlag.

zbelding · August 14, 2020, 12:50pm

To further reduce modem lock-ups in weak signal(which ultimately needs to be fixed by ublox), I have been doing this:

Open socket, send data
Close socket
Put the modem into AT+CFUN=0 mode
Before sending more data, put modem back to AT+CFUN=1 mode and re-attach

I will update my code in the google docs sometime this weekend.

Note: This strategy will not be very useful for very frequent updates, <10mins between updates.

zbelding · August 28, 2020, 12:47am

I decided to make a new link for this version. This would be for increased stability in weak signal areas:

----link removed, not needed with new modem firmware----

zbelding · August 30, 2020, 5:37pm

The version of this that I use for my application also sends alarm SMS via clickatell api. If anyone is interested, I can share this a well. In order to send sms reliably via clickatell, you need to purchase a phone number ($10/year) which they call a two-way integration. Other than that, it is free. I used it for a while without a two-way integration, but then my cell carrier started blocking the sms. Two-way integration insures the message goes through.

zbelding · September 4, 2020, 2:42pm

U-blox has released a new firmware update(L0.0.00.00.05.11, A.02.16) for this modem. One of the fixes listed in the release notes is the modem lock-up issue I have been battling:

[u-blox ID 5226]: Device crashing on live AT&T network in idle mode when SIB decode is
triggered by ML1 before finishing the Page decodes

I am testing this firmware now.

https://www.u-blox.com/en/docs/UBX-20033274

zbelding · September 9, 2020, 4:34pm

I have been running the new firmware for 5 days now. No modem lock-up issues, even while testing a omnidirectional antenna. I believe the new firmware has solved this issue. I am going to remove the second link I previously posted. It is not needed with the new firmware.

zbelding · September 16, 2020, 3:29pm

No modem lockups for almost two weeks. I would say this has cured the AT&T issues I have been dealing with.

ZeroSix · September 17, 2020, 10:37pm

I have the version L0.0.00.00.05.06,A.02.00 [Feb 03 2018 13:00:41] seems a bit hold, and I think is a bit relationed to my problem.

I tried to download the files but the link doest show nothing any ideas?

Topic		Replies	Views
MKR NB 1500 unable to connect to webserver MKR NB 1500	45	3361	May 9, 2023
MKR NB 1500 routinely disconnects after a week of transmission MKR NB 1500	19	3443	July 20, 2024
A working example of lower power and http requests MKR NB 1500	4	280	April 30, 2025
Trouble receiving SMS on MKR NB 1500 MKR NB 1500	46	4577	August 1, 2023
MKR NB 1500 modem help UDP and SMS MKR NB 1500	11	997	August 20, 2023

Sketch to http post to thingspeak and what I have learned so far

Related topics