[SOLVED] Recovering photos from SD card - Console freezes by big jpeg files

Hello there,
I had an "almost" dead SD card that my computer or camera wouldn't recognize but I checked the SD with Arduino's IDE example and it responded! Now I' am trying to extract the HEX code of each photo but my serial console freezes and I have to force quit Arduino IDE... that's because the photos are sized 10 - 15mb. To be sure that this method will work I inserted an SD card with a 100kb jpeg and it really worked. I extracted the HEX code from console and then I used a python script to convert the HEX into a jpg. All work fine when you deal with low sized jpgs but I need to deal with 10-15 mb jpegs to extract my "lost" photos... I'll post the code that I found in web to recover the HEX code of a photo and I would really pleased if you helped me out with it in case it needs a modification to let data flow and avoid the serial console freeze. Do I need to include a serial.flush() command? It's a buffer problem or what? I' am really new to Arduino projects and I can't figure out what the problem is. :~
Thanks in advance.

#include <SD.h>

File root;

void setup()
{
  Serial.begin(115200);
  pinMode(10, OUTPUT);

  SD.begin(10);

  root = SD.open("/");
  readDirectory(root, "");

  Serial.println("done!");
}

void loop()
{
  // nothing happens after setup finishes.
}

void readDirectory(File dir, String folder) {
  boolean files = true;
  while(files) {
    File entry =  dir.openNextFile();
    if (! entry) {
      files = false;
    } else {
      if (entry.isDirectory()) {
        String folder_new = folder;
        folder_new += entry.name();
        folder_new += "/";
        readDirectory(entry, folder_new);
      } else {
        outputFile(entry, folder);
      }
    }
  }
}

void outputFile(File entry, String folder) {
  Serial.print("--- ");
  Serial.print(folder);
  Serial.print(entry.name());
  Serial.print(";");
  Serial.println(entry.size(), DEC);
  byte r;
  while (entry.available()) {
    r = entry.read();
    if (r < 0x10) {
      Serial.print("0");
    }
    Serial.print(r, HEX);
  }
  Serial.println();
}

code and methodology found at SD card recovery using an Arduino | tiefpunkt tech

Now I' am trying to extract the HEX code of each photo but my serial console freezes

What "serial console" are you talking about?

Have you verified that the Arduino can even open a 15Mb file?

How long does it take to send a 15Mb file at 11,520 bytes per second?

What "serial console" are you talking about?

I 'm talking about the serial monitor of Arduino IDE.

Have you verified that the Arduino can even open a 15Mb file?

If I knew how, I would have tried it. Even with 1mb file size serial monitor freezes but this may be a problem of code and how it handles the process and not of arduino capabilities.

How long does it take to send a 15Mb file at 11,520 bytes per second?

Around 20mins if my calcs are correct. But I don't mind the time. I could leave it for some hours/days to get the photos if data flow properly.

In fact I just believe it's possible to transfer the file in HEX code no matter the size if you can pass the data directly from arduino to pc . Is this wrong? I just said I'm new with these projects and I need a serious answer about the serial connection and if it's doable to pass a 15mb file.Is there a problem with buffer? or should I include a flush command? I posted the code I've been using and I think there may be a modification to make it work properly.

In fact I just believe it's possible to transfer the file in HEX code no matter the size if you can pass the data directly from arduino to pc . Is this wrong?

Yes, it is. If you insert a 40 terrabyte SD card, the Arduino won't even recognize it. Size matters.

It's quite easy to add Serial.print() statement(s) to determine if the file is opened, to determine what the loop reading the file is doing, etc. While you are doing that, of course, the data being sent to the serial port can not actually be used to recreate the file, but, at least you learn where the process was failing, and then you could determine whether or not is was possible to fix that particular problem.

I know so far that the file is being opened and that's because I get the infos of jpeg and a part of the HEX code. For big files the serial monitor freezes after some time. I've given the code. Do you see any particular programming fault or bad use of serial lib? Maybe it's the buffer that it's full or...?

You are doing it wrong. Don't convert binary data to text. Write yourself a program that runs on the computer that gets the filename and the filesize and receives the raw data as is. Don't convert to and from text it makes the transfer over twice as slow. Please see Serial.write() - Arduino Reference
Also you are using SD.h please DO NOT USE SD.h instead use Google Code Archive - Long-term storage for Google Code Project Hosting.

Write yourself a program that runs on the computer that gets the filename and the filesize and receives the raw data as is.

So is it possible to write a sketch for arduino which will make it transfer the photos directly to a folder in my computer? And this only with sdfat and serial lib?

So is it possible to write a sketch for arduino which will make it transfer the photos directly to a folder in my computer?

No. The Arduino can only send data to the serial port on the PC. Some other application needs to be running that reads that data and does something with it.

No. The Arduino can only send data to the serial port on the PC.

Then, can I use processing language to extract the photos from arduino directly in a folder in my computer?

Then, can I use processing language to extract the photos from arduino directly in a folder in my computer?

No. You can use Processing to create an application that listens to the serial port that the Arduino is talking to, and have that application save the data to a file in a folder on your PC. Processing can not get the data from the SD card on the Arduino.

Ok. I tried to follow this path and I printed the binary code of a photo to serial monitor. After with some Processing code I managed to pass the binary from serial monitor to a .txt file hoping that if all go right I 'll rename it to .jpeg and I'll have the photo. But of course that didn't happen... My .txt file is 790kB when my photo is 490kB. I'll post the code and screenshots of monitors and I would be very pleased if you could give me a hand with this since I'am totally new and so confused!

  • print jpeg's binary on serial monitor
#include <SdFat.h>
SdFat sd;
SdFile myFile;
const int chipSelect = 10;

void setup() {
  Serial.begin(115200);
  Serial.println("Type any character to start");
  //while (Serial.read() <= 0) {}
  
  if (!sd.begin(chipSelect, SPI_FULL_SPEED)) sd.initErrorHalt();

  if (!myFile.open("formula.jpg", O_READ)) {
    sd.errorHalt("opening file for read failed");
  }
  Serial.println("formula.jpg:");

  
  int data;
  while ((data = myFile.read()) >= 0) Serial.write(data);

  myFile.close();
}

void loop() {
  // nothing happens after setup
}
  • read serial monitor with Processing and save binary in .txt file
// ReceiveBinaryData
import processing.serial.*;

Serial myPort; // Create object from Serial class
PrintWriter output;

short portIndex = 0; // select the com port, 0 is the first port
char HEADER = ':';


void setup()
{
// Open whatever serial port is connected to Arduino.
String portName = Serial.list()[portIndex];
println(Serial.list());
println(" Connecting to -> " + Serial.list()[portIndex]);
myPort = new Serial(this, portName, 115200);
output = createWriter("formula.txt");
}

void draw()
{
if( myPort.read() == HEADER) // after header is the jpg binary
{
  byte[] inBuffer = new byte[7];
  while (myPort.available() > 0) {
  inBuffer = myPort.readBytes();
  myPort.readBytes(inBuffer);
  if (inBuffer != null) {
  String value = new String(inBuffer);
  output.print(value);
  print(value);
    }
  }
 }
}

You should not be doing serial IO in draw(). Serial IO should be done in the serialEvent() callback.

  inBuffer = myPort.readBytes();
  myPort.readBytes(inBuffer);

Use ONE of these variations. You are throwing half the data away.

  String value = new String(inBuffer);
  output.print(value);

Pure horsecrap. You do NOT want to do diddly squat with text!

Pure horsecrap. You do NOT want to do diddly squat with text!

You're right but I'm not getting total garbage. I think there is a problem with the encoding ASCII or Unicode because when I change the file from Unicode to ASCII it's 488kB and the original is 491kB. Furthermore, the symbols in the begining are almost same but I'm losing some lines in the end of the file. :~
Check this for example:

Original: Ψΰ JFIF  H H  Ϋ C 
Received: �Ψ�ΰ JFIF  H H  �Ϋ C     
Original: Y_¤PΡ€Q,
Received: Y_¤PΡ�€Q,

Is there anything I can change to the above code to fix this problem? Plus the photos are 5Mb not 15 so with binary code I can really recover them within some hours. Any help would be appreciated. Thanks again.

I think there is a problem with the encoding ASCII or Unicode

Binary data is not encoded. So, it hardly seems likely to be an encoding issue.

Binary data is not encoded. So, it hardly seems likely to be an encoding issue.

I am talking about the way Processing code -who grabs the data from serial- writes them in the .txt file. Maybe there's the problem.

I am talking about the way Processing code -who grabs the data from serial- writes them in the .txt file. Maybe there's the problem.

JPEG data is typically not stored in text files. You REALLY need to understand binary vs. ASCII data. Neither end of the serial port should be attempting to understand the data.

Read a byte from the file. Write it to the serial port.

Read a byte from the serial port. Write it to a BINARY file using a BINARY stream writer.

DO NOT ATTEMPT TO INTERPRET THE DATA AS TEXT!

Thanks a lot for the guidance. I knew there was something wrong with my Processing code so I searched the Processing forum and I found this code that worked for me.

 import processing.serial.*;

    Serial myPort;
    OutputStream output;

     void setup() {

      size(320, 240);
      myPort = new Serial( this, Serial.list()[0], 115200);
      myPort.clear();

      output = createOutput("formula3.jpg");
    }

    void draw() {

      try { 
        while ( myPort.available () > 0 ) {
          output.write(myPort.read());
        }
      } 
      catch (IOException e) {
        e.printStackTrace();
      }
    }


    void keyPressed() {

      try { 
        output.flush();  // Writes the remaining data to the file
        output.close();  // Finishes the file
      } 

      catch (IOException e) {
        e.printStackTrace();
      }
    }

In combination with this code I modified for Arduino...

#include <SdFat.h>

SdFat sd;
SdFile myFile;
const int chipSelect = 10;

void setup() {
  Serial.begin(115200);
  //Serial.println("Type any character to start");
  //while (Serial.read() <= 0) {}
  
  if (!sd.begin(chipSelect, SPI_FULL_SPEED)) sd.initErrorHalt();

  if (!myFile.open("DCIM/###D####/_DSC####.JPG", O_READ)) {
    sd.errorHalt("opening file for read failed");
  }
  //Serial.println("formula.jpg:");

  
  int data;
  while ((data = myFile.read()) >= 0) Serial.write(data);

  myFile.close();
}

void loop() {
  // nothing happens after setup
}

... I finally managed to recover my 6Mb photos from my almost dead SD card. I only need to add an iteration to just don't do this manually using photo names. XD