Sketch to http post to thingspeak and what I have learned so far

I have made the following changes:

  1. I have removed the AT+CRSM commands used in the modemSetup function. These are probably not needed as discussed with @JohnJMaurer.

  2. I have reduced the delay between consecutive AT commands from 100ms to 10ms. The AT manual states it is wise to wait 100ms after receiving the final result code before sending the next AT command. I think this is in case there is a URC being sent after result code. This program only waits for result codes after specific commands, so I think 10ms is fine.

  3. I changed some of the timing of how the "+UUSORD:" URC is handled in the httpPost function.

I updated the google docs.

I have transitioned to an Arduino mega with the following:

And:

The mega better suits my application. I am using basically the same code with some different pin/uart assignments.

zbelding:
2. I have reduced the delay between consecutive AT commands from 100ms to 10ms. The AT manual states it is wise to wait 100ms after receiving the final result code before sending the next AT command. I think this is in case there is a URC being sent after result code. This program only waits for result codes after specific commands, so I think 10ms is fine.

I found the section of the AT manual that talks about this:
"The DTE should wait some time (the recommended value is at least 20 ms) after the reception of an ATcommand final result code or URC before issuing a new AT command to give the DCE the opportunity totransmit the buffered URCs. Otherwise the collision of the URCs with the subsequent AT command isstill possible."

I was wrong about a recommended 100ms delay. It is recommended to wait 20ms min. With this in mind, I have changed the delay to 25ms to avoid URC/AT command collision.

I have made the following changes:

I added "headers=false" to the http post body. This reduces the amount of headers in the thingspeak response.

In the sendCommand() function, I changed:
strchr(response, '@') ---> strstr(response, "\r\n@")
and
strstr(response, "ERROR") ---> strstr(response, "+CME ERROR")
This will reduce the likelihood of an "@" symbol or "ERROR" in an http response causing issues.

Also, I was not giving the response enough time to be fully buffered in Sara R4. Changing to a 250ms delay after the modem first tells me data is available seems to have helped the response come in with the first AT+USORD.

I added notes to the beginning of the sketch that talk about thingspeak talkback.

I have changed the following:

if (strstr(response, "200 OK") != NULL) {

to

if (strstr(response, "HTTP/1.1 200") != NULL) {

I learned that not all http servers reply with the "OK". Apparently only the "200" is part of the spec.

I have made the following change in the sendCommand() function:

if (loopCount > 100) {

to

if (loopCount > 2000) {

My mistake was in thinking that each main loop cycle would bring in a new character from the UART from the modem. In reality the processor is much faster than the serial data. This will reduce the number of times the response is searched via strstr() to check for “OK”, etc.

I also doubled the amount of time the rawHttpPost() function waits for the “+UUSORD:” from the modem telling us the server response has been buffered in the modem. I have seen it take longer than 30 sec when the cell signal is very poor:

    if (strstr(response, "+UUSORD:") != NULL) {
  delayStartTime = millis();
  sprintf(formattedCommand, "AT+USORD=%i,1024", socket);
  stage++;
}
if (millis() - delayStartTime > 30000) {
  stage = 0;
}

to

    if (strstr(response, "+UUSORD:") != NULL) {
  delayStartTime = millis();
  sprintf(formattedCommand, "AT+USORD=%i,1024", socket);
  stage++;
}
if (millis() - delayStartTime > 60000) {
  stage = 0;
}

Just a thought:
The MKRNB library normally cuts the data going into the socket into single packets of size 256.
But the modem can do more, see:
AT+USOWR=?
+USOWR: (0-6),(0-512), “HEX data”
+USOWR: (0-6),(0-1024), “data”
I have adjusted the MKRNB lib to
if (chunkSize > 512) {
chunkSize = 512;
because my MQTT messages are smaller than 512. 1024 should work too.
Since then, the packets go into the socket in one burst, which brought much more stability. I am not sure if this is the right direction. My pull: Enlarge the chunkSize by CptHolzschnauz · Pull Request #75 · arduino-libraries/MKRNB · GitHub
Please improve.
You do this manually with
sprintf(formattedCommand, “AT+USORD=%i,1024”, socket);
My experience so far regarding stability is not good with manual commands together with MKRNB lib. Actually it would be better if the MKRNB lib would work without errors …
The modem remains a damn tricky diva…

Arduino is probably just trying to reduce memory usage, but yeah you can read up to 1024 bytes from the modem buffer at once. It will only give you what is in the buffer. From the AT manual:

 //If the number of data bytes requested to be read from the buffer is bigger than the number of bytes
//stored in the buffer only the available amount of data bytes will be read.

Since the last two firmware updates, I have found the modem to be fairly stable. I have found that if you follow the AT manual and have proper response timeouts for each command(some can take 2-3 mins for a response) things work pretty well.

The problem with the Arduino lib is that it seems like they tried to take the GSM lib and modify it for this modem. LTE Cat M1 is a very different beast.

you are totally right, the code is almost identical, the modem(lte) is a beast diva,
the modem is actually much more stable since the new firmware (also my (denied by u-blox)sms problem has also suddenly disappeared)

About the 1024 byte : I use the MQTT lib from GitHub - 256dpi/arduino-mqtt: MQTT library for Arduino
with the Mosquitto Broker2.0.10 and i noticed that if i put data into the socket twice in a row the whole thing crashed. Since the data goes in in one go (also less overhead) it is stable.

Questions for you:

  • Do you do the communication with the modem all by yourself or do you also use the MKRNB lib? If both: Do you have problems in mixed mode?
    I have worked with modem which definitely does not work.

  • Could you explain the communication with the modem with str and sprintf a bit broader with your code?

    if (strstr(response, “+UUSORD:”) != NULL) {
    delayStartTime = millis();
    sprintf(formattedCommand, “AT+USORD=%i,1024”, socket);
    stage++;
    }
    if (millis() - delayStartTime > 60000) {
    stage = 0;
    }

How do you initialize the communication, how do you handle the responses etc. ?

Another hint I found out yesterday laboriously:
When changing the SIM, always cut the power from the mkr1500 or the modem will still talk half with the old SIM.

Would be happy about a little lesson from you. thx for your contributions so far, they were often enlightening…

I have made a bunch of changes that are too significant to list here, but here is a summary of it:

  • Added code that reads signal rsrq and outputs it to a field.
  • Removed Talkback, and in it’s place use another thingspeak field to send commands to the board.
  • A lot of changes to the section of the sketch that checks the http response. I am now reading the http header : Content-Length, and making sure the response body contains the same amount of bytes. This guarantees data is sent and received. It is especially helpful when reading commands from the command field discussed above. The major flaw with Talkback was that if the command was executed but the http response was lost, the arduino would never get the command.
  • I am not using the MKRNB lib at all.
  • strstr() is a C++ function that searches a char array for another char array. sprintf() is real handy to build char arrays from a combination of different variables types. I didn’t know about any of these until I found this great reference:
    https://www.cplusplus.com/reference/cstring/
  • The modem is setup using the modemSetup() function. All commands sent to the modem are sent with the sendCommand() function. A response timeout and a bool are sent with each command. The bool tells the function if you are going to accept a response error or not.

Thx for your explanations!

Voila! Now your solution makes sense for me.

I see…puhh…seems to be a lot of work and frustration to bear…
For other readers of this topic:
I found Mqtt publish to thingspeak without arduino library
where you can find zbelding’s riding school to bareback riding the modem beast…
What about a zbelding lib? I would already be a candidate for a change…

I added code to handle an issue where the the modem replies: “+USOWR: 0,0” to a AT+USOWR command. The second zero means the modem did not write any data. This section looks for the zero and starts the process over if it is found:

  if (stage == 21) {
    char *responsePointer;
    responsePointer = strstr(response, "+USOWR:");
    if (responsePointer != NULL) {
      if (responsePointer[10] - '0' != 0 ) {
        stage++;
      }
      else {
        stage = 0;
      }
    }
    else {
      stage = 0;
    }
  }

I also changed how far back http function resets if a partial response(or no response) is received. I was trying to reuse the active socket, but I ran into some issues with this because delayed data from the http server can still come into the modem buffer causing issues. I am now backing up to the beginning and opening another socket.

I added a section to modemSetup() to handle a sim card PIN if required. None of my cards need a pin, so I haven't fully tested.

Thanks for the great demonstration code. I'm encouraged to keep using my MKR NB 1500s now.

I know your aim wasn't to conserve power but if you did want to shut down the modem between Posts would you just PowerOff the modem, then PowerOn again (like in setup()) when you wanted to do the next Post? That is how I'm thinking of modifying your example. Any comments?

I think this would work well. I guess it depends on how often you are sending data. I will add some functionality to allow the modem to be powered off after each post. I will post here when I update the google doc.

I have added the ability to power down the modem after each post to conserve battery as suggested by @slicker . This will only make sense if the updates are not very frequent(once per hour or day). The reason being is that the modem has to re-register to the network upon power-up, which comes at a cost in terms of battery use.

The bool modemPowerDownAfterPost is the switch to use this feature.

The modem is powered off at the end of the rawHttpPost() function using AT+CPWROFF.

I removed the redundant modemPowerOn() function, and instead I made the modem reset function multipurpose. It can now just power on the modem or it can hard reset and power on the modem.

Hi @zbelding Thanks for producing a new version. I've tried it and think you'd be interested in my findings.
I've run it a couple of times and it just stops. The first time straight after restarting and the 2nd here
20:39:06.984 -> AT+USOCLCFG=1
20:39:06.984 ->
20:39:06.984 -> OK
20:39:06.984 ->
20:39:06.984 -> ATE0
20:39:07.031 ->
20:39:07.031 -> OK
20:39:07.031 ->
20:39:07.031 -> AT+CEREG?
20:39:07.078 ->
20:39:07.078 -> +CEREG: 0,2
20:39:07.078 ->
20:39:07.078 -> OK
20:39:07.078 ->
20:39:07.544 -> AT+CEREG?
20:39:07.589 ->
20:39:07.589 -> +CEREG: 0,2
20:39:07.589 ->
20:39:07.589 -> OK
20:39:07.589 ->
20:39:08.056 -> AT+CEREG?
20:39:08.056 ->
20:39:08.056 -> +CEREG: 0,2
20:39:08.056 ->
20:39:08.056 -> OK
20:39:08.056 ->

I'll reset and run it some more to see if there is a pattern. My NB 1500 has not had a firmware update, UBlox hasn't responded to my request for the file.

One thing I have noticed is the drop in signal. See the graph, the line of -9 is not powering off between Http Posts. The later ones are -9 or -20.
image
Previously the signal strength was all -8 or -9 with continuously powered on. Powering on again seems to start out with lower signal strength. That might be a cellular network thing.

Have you done some experiments or calculations to verify saving power by powering off the modem is only useful if the update frequency is > 1 hour? I'm surprised modem startup and connection is so power hungry.

Thanks again for the excellent work to improve NB 1500 behviour.

When you say it stops, I assume you mean the modem is repeatedly responding: +CEREG: 0,2. This means: "not registered, but the MT is currently trying to attach or searching an operator to register to". This is part of the re-registration process I referred to in post #57. It can take several minutes sometimes. The new firmware does help with this.

If you are updating every minute(which appears to be the case from your chart) I don't think powering off the modem is going to be a good solution for you. It would probably be best to use one of the modem power save modes that maintains registration to the mobile network. I will look into this.

I have zero data on this, I was just taking a wild guess.

With regard to the signal dropping to -20, this is probably because the modem has not been registered to the network long enough to calculate the signal.

Hi @zbelding ,

I should have been clearer. It stopped once in at +CEREG: 0,2 and the other time it was at "Modem on". Two different places. I ran it all night and it didn't stop again. Both times it did nothing for 15 minutes (no response at all), so I assumed it would be in that state forever.

I was updating every 60 seconds while continuously connected. I change the power off period to 10 minutes and restarted. The chart only showed 5 points at the 10 minute period.

I'll look into the current consumption over time to see how expensive a power down, power up and connect approach takes.