Issue read sd card large file and dump on computer using SerialUSB

Hi,

I wrote a sketch that read a file on sd card and write (send) it over SerialUSB.
The aim would be to download file present on the sd card to a computer using the SerialUSB (Arduino Due).
I searched the forum but I did not find precise information.

Environment:
Arduino Due
Sd card use SPI
SdFat release 21 March 2015

Here sketch code:

#include <SdFat.h>

#define INPUT_SIZE 64
#define BUF_SIZE 512

boolean bEseguiDump = false;
char IncomingString[INPUT_SIZE+1];
char FileName[INPUT_SIZE+1];

SdFat sd;

#define SD_CHIP_SELECT  77

//Dump data file to SerialUSB
void dumpFile (char* cFileName)
{
  SdFile file;
  uint8_t buf[BUF_SIZE];
  
  sd.vol()->cacheClear();
    
  if (!file.open(cFileName, O_READ))
  {
    SerialUSB.print("Error open file: ");SerialUSB.println(cFileName);
  }
  else
  {
    file.rewind();
    uint16_t i;
    
    i = file.read(buf , BUF_SIZE);
    while (i > 0)
    {
      SerialUSB.write(buf, i);
      SerialUSB.flush();
      memset(buf, '\0', sizeof(buf));
      i = file.read(buf , BUF_SIZE);
    }    
    file.close();
  }
}

void setup()
{
  memset (IncomingString, '\0', sizeof(IncomingString));
  memset (FileName, '\0', sizeof(FileName));
  
  while(!SerialUSB);
  SerialUSB.begin(115200);
    
  //See if the card is present and can be initialized:
  if (!sd.begin(SD_CHIP_SELECT, SPI_HALF_SPEED))
  {
    SerialUSB.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
}

void loop()
{
  if (SerialUSB)
  {
    while (SerialUSB.available())
    {
      byte iCntChar = SerialUSB.readBytes(IncomingString, INPUT_SIZE);
      IncomingString[iCntChar] = '\0';
      
      //Separo il comando e il nome del file
      //DUMP&NOMEFILE
      char *pch = strchr(IncomingString, '&');
      char cCommand[INPUT_SIZE+1];
      memset (cCommand, '\0', sizeof(cCommand));
      memset (FileName, '\0', sizeof(FileName));
      memcpy (cCommand, IncomingString, (pch-IncomingString) );
      strcpy (FileName, pch+1 );
      if (strcmp(cCommand, "DUMP" ) == 0)
      {
        if (strlen(FileName)>0)
          bEseguiDump = true;
      }
    }

    if (bEseguiDump)
    {
      dumpFile(FileName);
      bEseguiDump = false;
      memset (IncomingString, '\0', sizeof(IncomingString));
      memset (FileName, '\0', sizeof(FileName));
    }
  }
}

Here portion of c# code to read serial and write data on computer:

while (serialPort.BytesToRead > 0)
{
   byte[] SerialPortBytes = new byte[serialPort.BytesToRead];
   int LengthRead = serialPort.Read(SerialPortBytes, 0, SerialPortBytes.Length);
   iCntByte = iCntByte + LengthRead;
   fsDump.Write(SerialPortBytes, 0, SerialPortBytes.Length);
} //while byte

Looking dump file after read with WinHex, there are these issues:

  1. first read, dump file is truncated than the original on sd card, to be accurate lacking the last 3584 bytes (DumpFirstRead.bin)
  2. second read, dump file has 512 bytes more than original on sd card (DumpSecondRead.bin), dump file has at the beginning the 512 bytes that are absent in the first read.

Resetting Arduino Due and redoing the operations, they are repeat error listed above.

Why does this happen?

The dump file at the link:

DUMP FILE

Here bench.ino’s result of my sd card:

Manufacturer ID: 0X2
OEM ID: TM
Product: SA04G
Version: 1.1
Serial number: 0X2605F843
Manufacturing date: 12/2014

File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
167.79,152137,920,3022
174.45,153455,1614,2932
160.31,151939,1624,3182
160.14,151111,1614,3195
159.99,152373,1614,3188
161.04,151098,1615,3177
161.00,150890,1624,3177
161.08,151678,1615,3176
159.99,156401,1615,3198
160.91,150870,1614,3179

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
611.96,1486,680,835
612.03,1493,680,835
611.96,1497,680,835
612.03,1504,680,835
611.88,1465,680,835

Any suggestion is appreciated

My guess is that it's because you are relying on there being bytes available at the computer serial port every time your loop cycles around. If there is no byte available it appears that you are assuming that's the end of the file.

If you're going to reliably transfer files over serial I recommend you use a robust file transfer protocol designed to handle begin/end handshaking, error detection/correction, etc. I wrote an easy to use Arduino ZModem sketch here for example:

Announcing - Arduino SDCard file manager sketch with ZModem

To use it on the Due with SerialUSB, edit the definitions of DSERIAL and ZSERIAL in zmodem_config.h.

The DumpFile example in the SD section of the examples in the IDE works just fine on as Mega, and should be just as good on a Due. I'm not sure the size of the file is relevant, but my daily files were about 250k.

Thank all for response.

@monte_carlo_ecm, I'll try your advice.

@Nick_Pyner, I tried DumpFile and it's perfect for small files, but slow for large files (>1MB). Instead reading 512 bytes block on sd card file, with SdFat library and setting serialusb at 115200 baud, I was able to download data at about 320kbit/s. It seems that the sdcard at some point does not send more bytes on the SPI and a subsequent read request, the sd card starts with sending bytes from where it was previously blocked. I hope I have described good.

Do you think sd card overrun reading can create issues described above?

EDIT
320kB/s instead of 320kbit/s

I'm afraid I can't comment, as I don't know anything about the performance details.