Revisiting: SD file read to char*

travisstdenis · January 30, 2022, 7:33pm

I am trying to convert a BMP file into a char array so I can then convert that array into Base64. I found this old forum topic and implemented it exactly: https://forum.arduino.cc/t/sd-file-read-to-char/355318. It should be noted, this is 9kb .bmp file on an ESP8266 so RAM shouldn't be a concern.

Here is the full code:

#include <SPI.h>
#include <ESP8266WiFi.h>
#include <Adafruit_GFX.h>         // Core graphics library
#include <Adafruit_HX8357.h>      // Hardware-specific library
#include "SdFat.h"                // SD card & FAT filesystem library
#include "Base64.h"

using namespace sdfat;
#define SD_FAT_TYPE 1

SdFs sd;
FsFile dataFile;

#define USE_SD_CARD
#define TFT_CS   0
#define TFT_DC   15
#define SD_CS    2

Adafruit_HX8357        tft    = Adafruit_HX8357(TFT_CS, TFT_DC);

void setup(void) {
  Serial.begin(115200);

  tft.begin();          // Initialize screen

  Serial.println(F("Initializing filesystem..."));
  Serial.println("Initializing SD card...");
  if(!sd.begin(SD_CS)) {
    Serial.println(F("SD begin() failed"));
    for(;;); // Fatal error, do not continue
  }

  tft.fillScreen(HX8357_BLUE);


  delay(1000);
  char * pBuffer;
  String theFile = "wales.bmp";
  dataFile = sd.open(theFile, FILE_READ);
  if(!dataFile){
    Serial.println(F("File not found"));
    return;
  } else{
    Serial.println("File is open");
    unsigned int fileSize = dataFile.size();
    pBuffer = (char*)malloc(fileSize + 1);
    dataFile.read(pBuffer, fileSize);
    pBuffer[fileSize] = '\0';
    Serial.println(pBuffer); 
    Serial.println(sizeof(pBuffer));

    dataFile.close();
  }
}

void loop() {
// do nothing
}

this is the output on the Serial using the original code :
14:25:14.618 -> Initializing filesystem...
14:30:49.532 -> File is open
14:30:49.532 -> BM⸮
14:30:49.532 -> 4

It doesn't look like the file was fully converted as only the first three characters are shown on the Serial.println. I added in a Serial.println(sizeof(pBuffer)); and you can see it only says 4.

Why didn't the file convert? Is there another way?

Or is there a better way to get the BMP into an array to then run through Base64 encoding?

jremington · January 30, 2022, 7:44pm

Please edit your post to add your full code in line, using code tags.

travisstdenis · January 30, 2022, 11:27pm

I've since tried to do this explicitly and get different results. This is the snippet of the code that replaced the conversion part.

#include <SPI.h>
#include <ESP8266WiFi.h>
#include <Adafruit_GFX.h>         // Core graphics library
#include <Adafruit_HX8357.h>      // Hardware-specific library
#include "SdFat.h"                // SD card & FAT filesystem library
#include "Base64.h"

using namespace sdfat;
#define SD_FAT_TYPE 1

SdFs sd;
FsFile dataFile;


// Comment out the next line to load from SPI/QSPI flash instead of SD card:
#define USE_SD_CARD
#define TFT_CS   0
#define TFT_DC   15
#define SD_CS    2

Adafruit_HX8357        tft    = Adafruit_HX8357(TFT_CS, TFT_DC);

void setup(void) {
  Serial.begin(115200);

  tft.begin();          // Initialize screen

  Serial.println(F("Initializing filesystem..."));
  Serial.println("Initializing SD card...");
  if(!sd.begin(SD_CS)) {
    Serial.println(F("SD begin() failed"));
    for(;;); // Fatal error, do not continue
  }

  tft.fillScreen(HX8357_BLUE);

  delay(1000);
  String theFile = "wales.bmp";
  dataFile = sd.open(theFile, FILE_READ);
  unsigned int fileSize = dataFile.size();
  if(!dataFile){
    Serial.println(F("File not found"));
    return;
  } else{
    Serial.println("File is open");
    Serial.print("file size: "); Serial.println(fileSize);

    char pBuffer[fileSize + 1];
    Serial.print("pBuffer file size: "); Serial.println(sizeof(pBuffer));
    unsigned int incr = 0;
    char c;
    while(dataFile.available()){
      c = dataFile.read();
      Serial.print(c, HEX); Serial.print(" ");
      pBuffer[incr] = c;
      incr++;
    }
    pBuffer[fileSize] = '\0';
    Serial.println();
    Serial.print("incr value: "); Serial.println(incr + 1);
    
    dataFile.close();  
   }
}

void loop() {
  // do nothing
}

What happens is the output goes for a little bit then crashes and restarts. Here's the Serial error:

ets Jan  8 2013,rst cause:4, boot mode:(3,6)
wdt reset
load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00048660
~ld

I have no idea what's going on with either the original option or this one.

jremington · January 30, 2022, 11:34pm

pBuffer is a pointer, so sizeof returns 4.

What happens when you print "filesize"?

Unless the bitmap consists of printable ASCII characters, this line makes no sense. Serial.print() expects ASCII character strings, or numerical variables.

    Serial.println(pBuffer);

Always post ALL the code.

travisstdenis · January 31, 2022, 1:02am

when is do a

Serial.println(sizeof(pBuffer));

I get 4.

So then how do I print the pBuffer to see the results? How do I verify that the pBuffer was filled with the BMP's bytes because right now it does not look to be the case. Better yet, what if anything in the code needs to change to make it work? Would appreciate comments on both versions. I still need to encode it with Base64.

Always post ALL the code.

I didn't think it was necessary when only the relevant block changed.

jremington · January 31, 2022, 1:06am

We consider it necessary, because we know that people repeatedly make mistakes, as you clearly do in the "snippet". You are the one asking for help. From volunteers.

david_2018 · January 31, 2022, 1:26am

What format is the BMP file? Unusual to have a bitmap file made up of ASCII characters.

gfvalvo · January 31, 2022, 1:37am

Since a pointer occupies 4 bytes on an ESP8266 platform, that make perfect sense.

jremington · January 31, 2022, 1:41am

Binary, of course, with lots of unprintable characters and zeros to confound attempts to print them as if they are C-strings. Incidentally, the BMP file header contains, cleverly enough, the characters "BM", which explains this:

14:30:49.532 -> BM⸮

travisstdenis · January 31, 2022, 2:53am

Full code updated above

travisstdenis · January 31, 2022, 2:56am

Okay, so we've established that I have a pointer and that is 4 bytes. What I'm not reading is an explanation as to why the code - either option - are not giving me the expected results. I would like to know how to make the code work as I'd like in addition to pointing out the errors.

I appreciate the help of the volunteers helping novices likemyself.

jremington · January 31, 2022, 5:37am

Thank you for posting the full code. There are several technical problems with it. But first, you need to explain what this code is supposed to do.

What I'm not reading is an explanation as to why the code - either option - are not giving me the expected results.

What are those "expected results"?

A BMP image file is not a printable file. It contains raw image data to be sent to a display. Please explain why are you trying to print the file on the serial monitor.

Please also explain why you want to convert this file to Base64 representation.

travisstdenis · January 31, 2022, 1:59pm

As indicated in the first post, I'm trying to get a BMP image into a char array so that array can then be run through Base64 to upload to Google Drive. The printing was incidental, just to see if the conversion worked and the bitmap printed as a char string. My difficulty and where I was hoping to get some guidance is knowing if my two code options for conversion are correct and having a way to verify it.

gfvalvo · February 1, 2022, 1:46am

I wonder if it is, despite that bold claim. Your first code doesn't check if malloc() returned a null pointer. Your second code creates the array on the stack which probably lets you overwrite it into some other area.

I'd try something like (compiles, untested):

#include "Arduino.h"
#include <SdFat.h>
#include <memory>

using namespace sdfat;
#define SD_FAT_TYPE 1

void setup() {
  const uint8_t sdCs = 2;
  SdFs sd;
  FsFile dataFile;

  if (!sd.begin(sdCs)) {
    Serial.println("SD begin() failed");
    while (true) {
      yield();
    }
  }
  Serial.println("SD Started");

  dataFile = sd.open("wales.bmp", FILE_READ);
  if (!dataFile) {
    Serial.println("Unable to open file");
    while (true) {
      yield();
    }
  }
  Serial.println("File opened");

  size_t fileSize = dataFile.size();
  if (fileSize == 0) {
    Serial.println("File is empty");
    while (true) {
      yield();
    }
  }

  Serial.printf("File Size = %d\n", fileSize);

  std::unique_ptr<uint8_t[]> ptr { new (std::nothrow) uint8_t[fileSize] };
  if (ptr.get() == nullptr) {
    Serial.println("Unable to allocate array memory");
    while (true) {
      yield();
    }
  }
  Serial.println("Array allocated");

  uint8_t c;
  for (size_t i = 0; i < fileSize; i++) {
    c = dataFile.read();
    Serial.print(c, HEX);
    Serial.print(" ");
    ptr[i] = c;
  }

  dataFile.close();  
}

void loop() {
}

travisstdenis · February 1, 2022, 2:17am

I too reconsidered if i got ahead of myself on the memory available but as my previous test had showed that the file size of 8,420 is easily handled with the onboard 4mb of memory. Nonetheless, your code compiled and output the same as my second option. I did not understand this part of your code:

std::unique_ptr<uint8_t[]> ptr { new (std::nothrow) uint8_t[fileSize] };

Is it doing the same job as this from my code:

pBuffer = (char*)malloc(fileSize + 1);

I also noticed you did not have the trailing '\0'. I got the impression from many other topics on this forum that it was pretty essential.

gfvalvo · February 1, 2022, 2:08pm

You need a trailing null for ASCII c-strings. But, .bmp files are binary, right? So adding a trailing null makes no sense.

gfvalvo · February 1, 2022, 2:09pm

Your ESP8266 has 4MB of FLASH Memory, not RAM.

gfvalvo · February 1, 2022, 2:10pm

Please post the exact Serial port output for the code I suggested.

How about forgetting about the array for now and just try to read the file:

#include "Arduino.h"
#include <SdFat.h>

using namespace sdfat;
#define SD_FAT_TYPE 1

void setup() {
  const uint8_t sdCs = 2;
  SdFs sd;
  FsFile dataFile;

  if (!sd.begin(sdCs)) {
    Serial.println("SD begin() failed");
    while (true) {
      yield();
    }
  }
  Serial.println("SD Started");

  dataFile = sd.open("wales.bmp", FILE_READ);
  if (!dataFile) {
    Serial.println("Unable to open file");
    while (true) {
      yield();
    }
  }
  Serial.println("File opened");

  size_t fileSize = dataFile.size();
  if (fileSize == 0) {
    Serial.println("File is empty");
    while (true) {
      yield();
    }
  }

  Serial.printf("File Size = %d\n", fileSize);

  uint8_t c;
  for (size_t i = 0; i < fileSize; i++) {
    c = dataFile.read();
    Serial.print(c, HEX);
    Serial.print(" ");
  }

  dataFile.close();  
}

void loop() {
}

travisstdenis · February 2, 2022, 3:02am

the code from the initial post seems to work with the proper Serial.println(); commands. After the

dataFile.read(pBuffer, fileSize);

I run over it with this:

for(int i=0; i<fileSize; i++){
  Serial.print(pBuffer[i], HEX); Serial.print(" ");
}

And it spits out a long line of characters.

However, the code from post #3 still does not work (see the error message above) and I would like to know why. It looks as if I'm doing the exact same thing except not using a pointer char * for the pBuffer. It compiles and crashes.

system · August 1, 2022, 3:03am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Problem with SD card file dump(of BMP file) General Guidance	19	2133	May 5, 2021
SD Card bitmap read error/bug? Storage	10	6830	May 6, 2021
Writing binary file to SD (creating BMP files) Storage	8	16813	May 6, 2021
Weird issues using serial input with SD library Programming	18	5415	May 5, 2021
lettura file bmp da sd Software	1	370	May 7, 2021

Revisiting: SD file read to char*

Related topics