Help regarding conversion of jpg files to base64/hex/other format & viceversa

Hello Everyone! I need some help regarding image conversion. Is it possible to convert jpg image stored in the sd card of 640x480 size into base64/hex/other format such that I can place it in a string message to be sent from one arduino to another. I tried to read image file from SD card of arduino and the attached picture is what I get from the serial port (gibberish letters) which I cant process it as a string in arduino. I would really appreciate if someone could give me an example of how it is meant to be done. Please kindly reply. Thank you.

Screenshot by Lightshot Here is the picture.

If you communicate between arduinos, why do you want to convert to string?

With regards to the serial monitor, it only understands text. You can open your image file in a hex editor on the PC if you really need to see what's in there or use Serial.print(someVar, HEX).

read() is going to return a byte.
A byte can have a decimal value of 0 - 255.

The value of a byte can also be taken as representing a character. Here is an ASCII character table showing what characters each decimal value has been chosen to represent;

A lot of ASCII characters are not printable, they are control characters which are not readable by humans.
The readable characters are represented by decimal values 32 - 126 in the table.

If you want you can convert each byte you read into two printable characters representing hexadecimal digits (0...F).
You still will not be able to understand the data but will all be in the printing characters 0...9 and A...F.

@OP: What do you try to achieve? I'm almost sure that base64 or hex is not the way to go but I have to know what target you're trying to reach to help you. For sure the serial monitor is not able to show you a JPG image as it can only display text. If you encode the JPG in base64 you'll still see a lot of strange characters in the serial monitor. If you tell us more about your project we may have a solution for you.

sam_hudson1234:
Hello Everyone! I need some help regarding image conversion. Is it possible to convert jpg image stored in the sd card of 640x480 size into base64/hex/other format such that I can place it in a string message to be sent from one arduino to another. I tried to read image file from SD card of arduino and the attached picture is what I get from the serial port (gibberish letters) which I cant process it as a string in arduino. I would really appreciate if someone could give me an example of how it is meant to be done. Please kindly reply. Thank you.

Screenshot by Lightshot Here is the picture.

You REALLY need to study the file format for JPEG files. There is a WHOLE LOT MORE than just picture data.

Paul

sending/receiving aside.... WHERE are you trying to display this?

There has to be some 'end point' that you are trying to display this image at? Where is it?

Sending it to Arduino to Arduino......... to do what with it in the end?

When I have used an ESP8266 module to host/serve up some HTML web page... I converted some images to base64 encoding to place them 'in-line' with the HTML markup...

  • I only ever for 1 of the 2 images to show up.. (which ever was first)... never did work through it enough to see if it was a memory issues or something else. (was a Proof of Concept to show how easy it would be to use an ESP8266 module for identity theft for script kiddies)

My aim is to convert image jpg file into string such that i can transmit that string message between one LoRa transceivers. Previously I had done transmission between LoRa Transceivers with the help of string messages (char arr[] = "Hello World"). Text messages can be sent between lora transceivers not images. So my idea is to read and convert the image file jpg and then transmit between lora transceivers and then at the receiver end, i would like to recover the image back from the string after writing it to the image file. So what do you suggest me to do to achieve this?

sam_hudson1234:
My aim is to convert image jpg file into string such that i can transmit that string message between one LoRa transceivers. Previously I had done transmission between LoRa Transceivers with the help of string messages (char arr[] = "Hello World"). Text messages can be sent between lora transceivers not images. So my idea is to read and convert the image file jpg and then transmit between lora transceivers and then at the receiver end, i would like to recover the image back from the string after writing it to the image file. So what do you suggest me to do to achieve this?

Why convert to string? Just send bytes.

Paul

I want to send it as bytes but I need to put it in the following commands in the code

Serial.println("Sending to rf95_server");
// Send a message to rf95_server
uint8_t data[] = "Hello World!"; // this is where I want to put the image data that is read by arduino and into this particular data so that i can send it using the next command.
rf95.send(data, sizeof(data)); // this command only accepts uint8_t format only, so if there was character
array, i will need to use it as (uint8_t *) data.

rf95.waitPacketSent();

is it possible? if so, how can this be done?

I am sure your libraries support other data writing methods. What library are you using?

Paul

Libraries im using are PString, RH_RF95 for lora, and SD library.

Can you please provide links to the non-standard libraries (PString and RH_RF95). It's a bit much asked for us to dig through the web to find them (and possibly end up with the wrong versions if we manage to find them :wink: ).

If installed via the library manager, please indicate the versions that you installed.

When posting code, please use code tags

Type [code]
Paste your code after that
Type [/code] after the pasted code

Can you please post complete code for the code in reply #8. If it exceeds roughly 9000 bytes, you can attach.

I have used this code to read the image file from the sd card and I was able to read the image file but it shows gibberish on the serial monitor as seen in this link: Screenshot by Lightshot

#include <SD.h>   

File photoFile;   
String sentence;
 
void setup(){   
 
 Serial.begin(9600);   
 
 
 
 //Serial.println("initializing sd card");   
 pinMode(4,OUTPUT);     // CS pin of SD Card Shield   
 
 if (!SD.begin(4)) {   
  Serial.print("sd initialzation failed");   
  return;   
 }   
 
}   
 
 
void loop(){   

 
  Serial.flush();     
 
    File photoFile = SD.open("IMAGE01.JPG");   
      
     while (photoFile.available()) 
     {   
      sentence = photoFile.readStringUntil('\n');
      Serial.println(sentence);

     }   
     Serial.println(" ");
     Serial.println("ITS DONE");
     photoFile.close();   
     while(true) { ;;}
 
  }

Based on the picture, the data that you see, I want to read each line of the image file and encode it into the following commands that are used in RF95 to send and receive messages with each other. Here is the code from this website: RadioHead: rf95_client.pde

in that link there is a particular code i want to encode the image data in, which is as follows:

Serial.println("Sending to rf95_server");
  // Send a message to rf95_server
  uint8_t data[] = "Hello World!";
  rf95.send(data, sizeof(data));
  
  rf95.waitPacketSent();

I want to read the image data file from the sd card and put it in the data[] array such that it can be transmitted to another lora transceiver. How can this be done ?

want to read the image data file from the sd card and put it in the data[] array such that it can be transmitted to another lora transceiver. How can this be done ?

It can’t, there is not enough memory in the Arduino to hold an Array that big. A Uno only has 2K of memory for variables.

That code assumes the jpg file is in strings, that is not how jpg files are. They are simply an array of bytes. You seem to be ignoring all the replies you have been getting, why?

I didn't ignore brother, I did read all the replies, Its my first time posting in the forum so I may not be familiar with some options as to how to reply each person, so i replied at the end.

@sterretje: Well I want to communicate between arduinos via lora transceivers which is considered as a communication protocol between the arduinos (sender and receiver). I wanted to read the image file data that is stored in the sd card and then place it to uint8_t array like this (uint8_t data[]) so that I can send it via rf95 command: rf95.send(data, sizeof(data)). I cant use pc to open image file in hex editor because it has to be automated in the arduino itself.

@ardly : I know I can read the byte in hex format but I need to transfer the actual bytes from the file between lora transceivers rather than just the representation of it. I wanted to read the image file and place it onto the uint8_t data[] array so that I can send it via rf95.send(data, sizeof(data)) command Once the data is transferred on to the receiver lora, it should be written onto the image file such that I can open it via pc at the end. How can this be done?

@pylon + @xl97 : I have used this code to read the image file from the sd card and I was able to read the image file but it shows gibberish on the serial monitor as seen in this link: Screenshot by Lightshot

#include <SD.h>   

File photoFile;   
String sentence;
 
void setup(){   
 
 Serial.begin(9600);   
 
 
 
 //Serial.println("initializing sd card");   
 pinMode(4,OUTPUT);     // CS pin of SD Card Shield   
 
 if (!SD.begin(4)) {   
  Serial.print("sd initialzation failed");   
  return;   
 }   
 
}   
 
 
void loop(){   

 
  Serial.flush();     
 
    File photoFile = SD.open("IMAGE01.JPG");   
      
     while (photoFile.available()) 
     {   
      sentence = photoFile.readStringUntil('\n');
      Serial.println(sentence);

     }   
     Serial.println(" ");
     Serial.println("ITS DONE");
     photoFile.close();   
     while(true) { ;;}
 
  }

Based on the picture, the data that you see, I want to read each line of the image file and encode it into the following commands that are used in RF95 to send and receive messages with each other. Here is the code from this website: RadioHead: rf95_client.pde

in that link there is a particular code i want to encode the image data in, which is as follows:

Serial.println("Sending to rf95_server");
  // Send a message to rf95_server
  uint8_t data[] = "Hello World!";
  rf95.send(data, sizeof(data));
  
  rf95.waitPacketSent();

I want to read the image data file from the sd card and put it in the data[] array such that it can be transmitted to another lora transceiver. How can this be done ?

Sam, put the SD card in a PC and use a hex editor to view the file. Then you can see with your own eyes what you are dealing with. Stop the guessing!

Paul

You can start with something like below; it's a (slightly) modified version of Using the SD library to read and write to a file on a SD card.

#include <SPI.h>
#include <SD.h>

File photoFile;
char fileName[13] = "IMAGE01.JPG";
void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(57600);
  while (!Serial)
  {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  Serial.print("Initializing SD card...");

  if (!SD.begin(4))
  {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");

  // open the file for reading:
  photoFile = SD.open(fileName);
  if (photoFile)
  {
    Serial.println(fileName);

    // read from the file until there's nothing else in it:
    while (photoFile.available())
    {
      // read byte
      byte data = photoFile.read();
      // print to serial monitor
      Serial.print(data, HEX);
      // print a separator
      Serial.print(" ");
      // transmit; just based on the code snippet that you provided
      //rf95.send(&data, 1);
      //rf95.waitPacketSent();
    }
    // close the file:
    photoFile.close();
  }
  else
  {
    // if the file didn't open, print an error:
    Serial.println("error opening file");
  }
}

void loop()
{
}

The code prints the contents of the file to serial monitor (one long line) and (commented out) shows how to send; the latter based on the code you provided. You can add your radio setup stuff to the code, uncomment the rf95 lines in above code and start transmitting.

Now this is highly inefficient, sending one byte at a time. I don't know how many bytes you can send at a time with the radio library so lets do a conservative approach with 16 bytes,

#include <SPI.h>
#include <SD.h>

// size of radio tx buffer
#define TXBUFFERSIZE 16

// file handle
File photoFile;
// file to open
char fileName[13] = "IMAGE01.JPG";
// buffer for radio transmission
byte txBuffer[TXBUFFERSIZE];

void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(57600);
  while (!Serial)
  {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  Serial.print("Initializing SD card...");

  if (!SD.begin(4))
  {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");

  // open the file for reading:
  photoFile = SD.open(fileName);
  if (photoFile)
  {
    Serial.println(fileName);

    // read from the file until there's nothing else in it
    while (photoFile.available())
    {
      // read up to 16 bytes; returns the actual number of bytes that were read or an error code
      int16_t numBytes = photoFile.read(txBuffer, sizeof(txBuffer));
      if (numBytes < 0)
      {
        // inform user about read error
        Serial.print("Error while reading file; error number ");
        Serial.println(numBytes);
        // bail out.
        break;
      }

      // print each byte that was read from file to the serial monitor, seperated by a space
      for (int16_t dataCnt = 0; dataCnt < numBytes; dataCnt++)
      {
        Serial.print(txBuffer[dataCnt], HEX);
        Serial.print(" ");
      }

      // transmit; just based on the code snippet that you provided
      //rf95.send(txBuffer, numBytes);
      //rf95.waitPacketSent();
    }
    // close the file:
    photoFile.close();
    // inform user that we're done
    Serial.println();
    Serial.println("Done");
  }
  else
  {
    // if the file didn't open, print an error:
    Serial.println("error opening file");
  }
}

void loop()
{
}

Now there are three things outstanding (my opinion).

Before sending the file, it's advisable to inform the other side how many bytes are to be expected. I'm not quite sure what is possible (no card reader connected to an Arduino at this moment) but Using the SD library to print the directory of files on SD card might be a start.

Before sending each packet, tell the receiver how many bytes are in the packet. That will allow the receiver to check if the correct number of bytes are received and possibly send a reply (no idea about the radio, does it support two-way communication?) back to the transmitter if something is wrong (e.g. use a timeout if the expected number of bytes is not received within N seconds). It can also confirm correct reception.

Because radio transmission is prone to external influences, I would add a CRC after sending all bytes or after sending a 'packet'. This will allow the receiver to check if the data was corrupted and request e.g. a resend.

1)
determine filesize
2)
send filesize (2 bytes)
  wait for reply from receiver to confirm
3)
read bytes from card; add each byte to crc
4)
send up to 16 bytes
  wait for reply from receiver to confirm
5)
send crc
  wait for reply from receiver to confirm

I suggest that you start with a text file to test everything; it's easier to verify.