Misinterpreted NULL terminator

Projects Programming
1

This here it is more difficult than sweeping stair from bottom to top

I have a TXT file recorded in SPIFFS. In it I write lines with date and time. That's all, date and time.

I try to read this file with another piece of code which is this one, to encode it:

void base64()  {

  myFile = SPIFFS.open("/DATAHORA.txt", "r");
  if (myFile) {
      unsigned int fileSize = myFile.size(); 
      uint8_t * pBuffer = (uint8_t *) malloc(fileSize);  
      if (pBuffer != NULL) {
        myFile.read(pBuffer, fileSize);
        String toEncode =(char*)pBuffer;
        encoded = base64::encode(toEncode);
      free(pBuffer);                        
      myFile.close();
     }
  }
  
}

But because of this NULL (I already tried nullprt, \n, \0, etc), I only encode the first line of the file. The other lines are ignored.

The timestamp lines in the file were formed using this, note that only the last one contains the println command.

myFile.print(now.Day() < 10 ? "0" : "");
      myFile.print(now.Day(), DEC);
      myFile.print('/');
      myFile.print(now.Month() < 10 ? "0" : "");
      myFile.print(now.Month(), DEC);
      myFile.print('/');
      myFile.print(now.Year(), DEC);
      myFile.print(',');
      myFile.print(now.Hour() < 10 ? "0" : "");
      myFile.print(now.Hour(), DEC);
      myFile.print(':');
      myFile.print(now.Minute() < 10 ? "0" : "");
      myFile.print(now.Minute(), DEC);
      myFile.print(':');
      myFile.print(now.Second() < 10 ? "0" : "");
      myFile.println(now.Second(), DEC);

But each line has its null terminator because when I read the SPIFFS file I see correctly the lines one below the other.

So how to solve it ?

2

if you have the Serial monitor open, can you replace your function with

void base64()  {
  myFile = SPIFFS.open("/DATAHORA.txt", "r");
  if (myFile) {
    size_t fileSize = myFile.size();
    uint8_t * pBuffer = (uint8_t *) malloc(fileSize + 1);
    if (pBuffer != nullptr) {
      myFile.read(pBuffer, fileSize);
      pBuffer[fileSize] = '\0';
      Serial.print("This is what's in the buffer now");
      Serial.println((char *) pBuffer);
      free(pBuffer);
    } else {
      Serial.println("could not allocate enough memory");
    }
    myFile.close(); // <=== BTW careful on where you place this
  } else {
    Serial.println("could not open the file");
  }
}

and show us what you get

3

Empty, there's nothing below this line:

Serial.print("This is what's in the buffer now");
There's nothing here

But look, I just downloaded the .TXT file from the web server. And it's normal.

DataHora
DataHora700×463 56.1 KB

One fact that caught my attention is that before I could open the file on the webserver itself, then suddenly I can't open it anymore, I can only download it.

As if the file was made read-protected. Now we see that that initial piece of code is also no longer able to read the file.

4

\n is the newline control code, not a null.

5

Using '\n' gives this error when compiling. pBuffer is uint8_t:
invalid conversion from 'uint8_t* {aka unsigned char*}' to 'const char*' [-fpermissive]

6

can you try again with

void base64()  {
  myFile = SPIFFS.open("/DATAHORA.txt", "r");
  if (myFile) {
    size_t fileSize = myFile.size();
    Serial.print("file size: ");  Serial.println(fileSize);
    uint8_t * pBuffer = (uint8_t *) malloc(fileSize + 1);
    if (pBuffer != nullptr) {
      myFile.read(pBuffer, fileSize);
      pBuffer[fileSize] = '\0';
      Serial.print("This is what's in the buffer now");
      Serial.println((char *) pBuffer);
      free(pBuffer);
    } else {
      Serial.println("could not allocate enough memory");
    }
    myFile.close(); // <=== BTW careful on where you place this
  } else {
    Serial.println("could not open the file");
  }
}
7

OK

file size: 791
This is what's in the buffer nowQuarta,22/02/2023,13:22:55

It just returned the first line of the file confirming my first topic, it's just throwing the first line into the buffer.

8

I deleted and recreated the file and I think it will work now. I'll be right back with this information.

9

file size: 108
This is what's in the buffer nowJQuarta,22/02/2023,16:54:58
Quarta,22/02/2023,16:56:38
Quarta,22/02/2023,16:58:04

I just deleted and created again.

Thanks a lot for your time and help.

10

Is there any test to eliminate this 'dirt'? always generated by the last 20 characters of the base64 string

BaseToTXT

The code:

myFile = SPIFFS.open("/DATAHORA.txt", "r");
  if (myFile) {
      //encoding base64
      unsigned int fileSize = myFile.size();             // Get the file size.
      uint8_t * pBuffer = (uint8_t *) malloc(fileSize);  // allocate memory for the file.  
      if (pBuffer != nullptr) {                          // if allocation worked
        myFile.read(pBuffer, fileSize);                  // then read the whole file into SRAM.
        String toEncode =(char*)pBuffer;
        encoded = base64::encode(toEncode);
        Serial.println(encoded);
      free(pBuffer);                                     // Free the memory that was used by the buffer.      
     }
     myFile.close();
  }
11

Compare this

With my code

When you read the file, you don’t add the null char termination and so when you build the String it read the memory until it finds a zero. If you are unlucky and memory is not null then you get extra characters in the String.

In my code I make room for one extra char and manually use it at the end of the buffer and set it to 0. This way the c-string conversion to a String (what a waste of memory) will work

12

Brilliant thought.

Worked perfectly.

Health and peace.

13

It's a colossal waste of memory. Especially when you consider there is an overload of the encode() function that doesn't require making a superfluous extra copy of the input data in a String object:

String base64::encode(const uint8_t * data, size_t length)
{

Moreover I wonder about the point of using BASE64 for a text file in the first place. The purpose of BASE64 is to allow representation of binary data as printable ASCII characters. For this ability, it extracts a penalty of 33% expansion in the data size over the original binary. However, that's better than the 100% expansion paid for using ASCII Hex. But, using it for a text file (which is already printable) has me puzzled.

14

I supposed the recipient side always expects base64 and decodes it

15

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.