ESP32, FONA and MQTT

Hi everyone,

For this project I'm using an ESP32, connected to a FONA SIM800H board. ( Rx, Tx and RST ). I'm using the latest versions of Adafruit_FONA and Adafruit_MQTT. My MQTT-Broker is Mosquitto on Linux and all seems to work. Except......

I'm creating an XML-String with 447 bytes and the FONA reports that "only" 128 bytes are being sent. MQTT explorer shows that only a portion is received by the broker. ( Other Linux / C++ based software is working perfectly.. )

Is there a limit? Someone else had this behaviour? I've been looking at the MAX_BUFFER of 150, defined in Adafruit_MQTT.h and tried to increase this to 1024. My expectation would be that this should be sufficient, but to no avail.. I've noticed in the

uint16_t Adafruit_MQTT::publishPacket

there is a magic number 128. ( Adafruit_MQTT.cpp, around Line 790 and 791) Is there anyone who can explain this number to me? ( The Serial.print statements are added by me to get some idea of what's happening.. )

// as per
// http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718040
uint16_t Adafruit_MQTT::publishPacket(uint8_t *packet, const char *topic,
                                      uint8_t *data, uint16_t bLen, uint8_t qos,
                                      uint16_t maxPacketLen) 
{
      uint8_t *p = packet;
      uint16_t len = 0;

      // calc length of non-header data
      len += 2;             // two bytes to set the topic size
      len += strlen(topic); // topic length
      Serial.print( "Topic Length    : " ); Serial.println( len );

      if ( qos > 0 ) 
      {
          len += 2; // qos packet id
      }
      // Calculate additional bytes for length field (if any)
      uint16_t additionalLen = packetAdditionalLen(len + bLen);

      Serial.print( "Additional Length  : " ); Serial.println( additionalLen );

      // Payload remaining length. When maxPacketLen provided is 0, let's
      // assume buffer is big enough. Fingers crossed.
      // 2 + additionalLen: header byte + remaining length field (from 1 to 4 bytes)
      // len = topic size field + value (string)
      // bLen = buffer size
      if (!(maxPacketLen == 0 ||
            (len + bLen + 2 + additionalLen <= maxPacketLen))) {
        // If we make it here, we got a pickle: the payload is not going
        // to fit in the packet buffer. Instead of corrupting memory, let's
        // do something less damaging by reducing the bLen to what we are
        // able to accomodate. Alternatively, consider using a bigger
        // maxPacketLen.
        bLen = maxPacketLen - (len + 2 + packetAdditionalLen(maxPacketLen));
      }
      len += bLen; // remaining len excludes header byte & length field

      // Now you can start generating the packet!
      p[0] = MQTT_CTRL_PUBLISH << 4 | qos << 1;
      p++;

      // fill in packet[1] last
      do 
      {
            Serial.println( "[=========================================================]" );
            Serial.print( "len before   : " ); Serial.println( len );
            uint8_t encodedByte = len % 128;
            len /= 128;

            Serial.print( "encodedByte  : " ); Serial.println( encodedByte );
            Serial.print( "len after    : " ); Serial.println( len );
            // if there are more data to encode, set the top bit of this byte
            if ( len > 0 ) 
            {
              Serial.print( "encodedByte          : " ); Serial.println( encodedByte );
              encodedByte |= 0x80;
              Serial.print( "encodedByte |= 0x80  : " ); Serial.println( encodedByte );
            }
            p[0] = encodedByte;
            p++;
      } while (len > 0);
      Serial.println( "[=========================================================]" );

      // topic comes before packet identifier
      p = stringprint(p, topic);

      // add packet identifier. used for checking PUBACK in QOS > 0
      if (qos > 0) 
      {
        p[0] = (packet_id_counter >> 8) & 0xFF;
        p[1] = packet_id_counter & 0xFF;
        p += 2;

        // increment the packet id
        packet_id_counter++;
      }

      memmove(p, data, bLen);
      p += bLen;
      len = p - packet;
      DEBUG_PRINTLN(F("MQTT publish packet:"));
      DEBUG_PRINTBUFFER(buffer, len);

      Serial.print( "Buffer Length  : " ); Serial.println( len );

      return len;
    }

    uint8_t Adafruit_MQTT::subscribePacket(uint8_t *packet, const char *topic,
                                           uint8_t qos) {
      uint8_t *p = packet;
      uint16_t len;

      p[0] = MQTT_CTRL_SUBSCRIBE << 4 | MQTT_QOS_1 << 1;
      // fill in packet[1] last
      p += 2;

      // packet identifier. used for checking SUBACK
      p[0] = (packet_id_counter >> 8) & 0xFF;
      p[1] = packet_id_counter & 0xFF;
      p += 2;

      // increment the packet id
      packet_id_counter++;

      p = stringprint(p, topic);

      p[0] = qos;
      p++;

      len = p - packet;
      packet[1] = len - 2; // don't include the 2 bytes of fixed header data
      DEBUG_PRINTLN(F("MQTT subscription packet:"));
      DEBUG_PRINTBUFFER(buffer, len);
      return len;
}

{EDIT} It is of no use to publish the entire code here. The entire library is hosted on github..

The library you're using (Adafruit_MQTT) has a buffer size (holding the complete MQTT packet) of 150 bytes. So your payload must be considerably smaller than this (at least using this library).
Every MQTT library I'm aware of has an upper limit of the packet size (PubSubClient: 256). Of course you can change that packet size limit in the library but I would think about your concept. Using MQTT to transfer larger XML strings is probably the wrong technology. The clever thing in MQTT is that some client publishes some values and another client subscribes exactly the values it's interested in. For XML content I prefer HTTP as the transfer method.

Thank you for your reply. I do understand the idea of small messages, but this sensor should deliver data to an already existing infrastructure. Instead of JSON the encoding is ( unfortunately ) XML. I should figure out a way to adapt the used library or use our own MQTT lib and get it working with the FONA instead of WiFi....

I think we have it solved. We were focussing on the Adafruit_MQTT classes without looking at the Adafruit_FONA class.

Everything seemed ok for sending large packages, but still the broker only received a part of the message. Turned out the Length of the buffer ( In our case 501 bytes ) was reported back as 247. Which is a bit strange. Drilling deeper we found the actual method for sending raw-TCP packages in the Adafruit_FONA library. Length is give there as a uint8_t which has a maximum of 255.... Changing it to uint16_t in the header and implementation solved the limiting issue.

I'll put in a pull-request at adafruit.

Thx everybody who read this. Hopefully this will be of use for you in later projects.