Substring is not working with .bin file retrieving from serial of sim module a7672s

[anantpsi123]

Hi team,

I'm trying to do ota using simcom a7672s , using AT command set, code is getting .bin file successfully, but when i am trying to do same OTA with at command ... as below

1. AT+HTTPPARA="URL","firmware.bin"
2. at+httpaction=0
3. At+httpread=0, content-length got using at+HTTPHEAD

But this httpread command taking buffer of 1024 byte only ...
So its recieving .bin file with HTTPREAD :1024 .... .bin file data.... HTTPREAD: 0

So i planned to use substring command but ... Substring only taking 10 or 14 byte out of 1000+ byte long string.

Actually I printed .bin file all byte during recieved by serial1 and
Then as usual process of ota, saving these all same byte to flash memory.

[namanshandilya]

i also facing similar issue

[soumil696969]

I am also having the same issue if you find the solution, kindly help! Your help will be appreciated.

[sonofcy]

@namanshandilya @soumil696969 @anantpsi123 So you all go to the same school?

[J-M-L]

@anantpsi123

So i planned to use substring command but ... Substring only taking 10 or 14 byte out of 1000+ byte long string.

substring() works for the String class... you have a binary file.... (a c-String is null terminated so if you try to load a binary buffer into a String, it will stop at the first null byte it finds)

don't use any String related function. Go bare metal with bytes and arrays and associated functions like memcpy() or memmove()

[christop]

@J-M-L

substring() works for the String class... you have a binary file.... (a c-String is null terminated so if you try to load a binary buffer into a String, it will stop at the first null byte it finds)

This is where the Arduino's String still falls very short of a C++ std::string and where I would log bugs against Arduino. A std::string can hold any arbitrary data, including null bytes, but a String still (inconsistently) acts like a null-terminated C string.

[J-M-L]

agreed - that's a difference.

it would be interesting to know how OP was building the String from the payload

[anantpsi123]

Hey,

Thanks for the suggestion!

I am using a file system to process my data, and I’ve already implemented a method similar to what you suggested. However, I’m still facing an issue where certain unwanted patterns, such as +HTTPREAD:, remain uncleaned from the data.

Here’s the code I’m currently using:

const char *unwantedPatterns[] = {
  "OK",
  "AT+HTTPREAD=0,1024",
  "+HTTPREAD: 1024",
  "+HTTPREAD: 0",
  "+HTTPREAD: 268"
  "+HTTPREAD:",
  "ERROR",
  "PB DONE",
  "\n",
  "\r",
};
const size_t numPatterns = sizeof(unwantedPatterns) / sizeof(unwantedPatterns[0]);

if (myFile) {
    Serial.println("Reading firmware.bin...");
    myFile1 = SD.open("cleanFirmware.bin", FILE_WRITE);

    if (!myFile1) {
      Serial.println("Error opening cleanFirmware.bin");
      myFile.close();
      return;
    }

    const size_t bufferSize = 512;  // Process in chunks of 512 bytes
    uint8_t buffer[bufferSize];
    size_t bytesRead;

    while ((bytesRead = myFile.read(buffer, bufferSize)) > 0) {
      size_t cleanIndex = 0;

      for (size_t i = 0; i < bytesRead; i++) {
        bool patternMatched = false;

        // Check for unwanted patterns
        for (size_t j = 0; j < numPatterns; j++) {
          size_t patternLength = strlen(unwantedPatterns[j]);

          // Ensure the pattern can fit in the remaining buffer
          if (i + patternLength <= bytesRead && memcmp(&buffer[i], unwantedPatterns[j], patternLength) == 0) {
            i += patternLength - 1;  // Skip the pattern
            patternMatched = true;
            break;
          }
        }

        // If no pattern matched, copy the byte to the clean buffer
        if (!patternMatched) {
          buffer[cleanIndex++] = buffer[i];
        }
      }

      // Write the cleaned chunk to the output file
      myFile1.write(buffer, cleanIndex);
    }
}

Here’s an example of the data I received after processing it with this code:

Ñãy¢yeC@+ ÙwçŒ"rCH2ªªç+щ3sC yâx¡xäyH3$Cë @, Ù@ €² ) Ð
,ÙxŒBÑ
8
3€² 0 ü÷Tþ
 Rç
+ûÑ‹3sCáxëYqK +Ð0  à ¿ ( ÑAç! 8 ÿ÷Òûèç
+æÑŒ"rC2Áç
+àÑŒ"rC2»ç
+ìÐ+Øщ3sC¡xíK)q +ÐÐ0  à ¿ÌçÀFœ          sµeL  xbxˆBБBÐ" Ž4 x2xˆBÐ  ŠBÑ
 L à  bxŠB	ÑJ *Ð! š²1 à ¿
 v½¢xŠBúÑ&  "H6›²  “3 ü÷êý@#2 ( ¡xÿ÷2øëçÀFœ  (      Kpµ %04%p J<4p™hškŠšb$x ,Ñ
M‡ >5,p )Ð *Ð{Ò	Ð.3xÿ÷;øp½/3ùçô        Kpµ L +Ð`{ à ¿ (ÑK
!£‡>4 p( ÿ÷Ãÿp½£h¢k›@+ Ù@#/4( J!x›²þ÷ßÿðç    ô 
+HTTPREAD: 1024
  '  ´  ðµ(L እh‰° )ÐIº‰²(  ð
ü²"i#}§keCeº8 “ ðmüK “›íeÁ`{@- Ù@% •›ü÷býÚK
!£‡ #>40 #pÿ÷zÿ	°ð½.4$x (Ñ
  «"Àþ÷yü #¯) 8 >p{p<q • ðüç烲! 0 šþ÷†ÿàçÀFô  ´  :  Ã}‚}eCBz
zCz	CÂzCÑ C˜A@ pGÁ{()Ñ
{ É	÷Ð
  +ôÐ[º›²šB’ARBP íç*)óÑ
{ É	ïÐæçµ@" !Hþ÷.ü½ÀFô  µ@" !Hþ÷$ü½ÀFô  sµKy +
Ð  v½‹y+úÑËyP+÷Ñ*õÙŠxKNp
x –É

As you can see, some patterns like +HTTPREAD: are still present in the data.

Do you have any additional suggestions to refine this process and fully clean the data?

Much appreciated!

[J-M-L]

Your pattern matching won’t work if the buffer you loaded cuts the pattern

As an example, Say your file holds ABABXYZABA and your pattern is XYZ and you read in chunks of 5 bytes
The first buffer is ABABX and does not have the pattern, the second buffer is YZABA and does not have the pattern . Yet the file had the pattern

Seems you saved some header info in the file - that should not be done. You should get rid of that header as you receive the incoming data and get a clean bin file.

[anantpsi123]

Thank you for pointing that out. I understand the issue now and will work on implementing a solution to ensure that the pattern matching handles scenarios where the pattern might be split across buffers. Additionally, I'll make sure to discard any unnecessary header information as the data is received, ensuring a clean binary file for processing.