Downloading .bin from FTP and saving to SD CARD

sergioarduino · April 5, 2023, 2:41pm

I have a file called actual.bin (195Kb) which is just the blink.ino example.

By filezilla I access this file through FTP.

I would like to save it in the SD CARD module that I have connected to my ESP32. It's just testing.

I used this block but it created a 64Kb UPDATE.bin file on the SD CARD. The original is 195 Kb.

The code contains errors because it doesn't have a *char buffer in it, and I think that's what's missing.

But I've been searching for hours and I can't find similar codes. It's just that if I go by theory I won't know how to do it because I lack a lot of information about this science of programming.

Can anyone paste a code snippet that works or fix this code snippet of mine ?

myFile = SD.open("atualiza.bin", FILE_WRITE);
   if (myFile) {
     ftp.OpenConnection();
     ftp.ChangeWorkDir("/htdocs");
     String response = "";
     ftp.InitFile("Type A");
     ftp.DownloadString("atualiza.bin", response);
     myFile.println(response);
     delay(100);
     myFile.close();
     }

Post a "try this":

Thanks

sergioarduino · April 5, 2023, 7:09pm

If I use a .TXT file in the above code it works. I tested it with a 25Kb file containing 1000 lines, what I found strange is that the copied file came with 1 blank line after each line of the original file. It might have something to do with CR+LF, or something like that.
But I didn't see where to solve it.

For the .BIN I put ftp.InitFile("Type I");
But even so I brought a 64Kb .BIN and not the 195Kb of the original file that is there on the server.

I'm using this library: <ESP32_FTPClient.h>

Does anyone know how to fix it ?

...

anon57585045 · April 5, 2023, 7:12pm

If there is a compiler error listing you should post the whole thing.

anon57585045 · April 5, 2023, 7:14pm

You've been around the forum for a while, right? Have you missed that this is explicitly discouraged?

qubits-us · April 5, 2023, 7:18pm

probably shouldn't try downloading the firmware into a string var..

void DownloadFile(const char * filename, unsigned char * buf, size_t length, bool printUART = false);

this would be better, send it in a buffer and size, buffer should be size of firmware..
good luck.. ~q

sergioarduino · April 5, 2023, 7:19pm

Compiles without error.
Copies the .BIN file from the server to the SD CARD.
But it copies only a piece of it, 64 Kb and not all of it, 195 Kb.
I'll post the whole code.

sergioarduino · April 5, 2023, 7:22pm

#include "Arduino.h"
#include <WiFi.h>
#include <WiFiClient.h> 
#include <ESP32_FTPClient.h>
#include <SPI.h>
#include <mySD.h>

const char* ssid = "zzzzzzzzzzzzzzzz";
const char* password = "zzzzzzzzzzzzzzzz";

char ftp_server[] = "ftpupload.net";
char ftp_user[]   = "ssnh_33564483";
char ftp_pass[]   = "Singian@01";

ESP32_FTPClient ftp (ftp_server,ftp_user,ftp_pass);

int semwifi;

#define CHIPSELECT   5
#define MOSI        23
#define MISO        19
#define CLOCK       18

File myFile;

void setup()
{
  Serial.begin(115200);
  delay(500);
  
  if (!SD.begin(5, 23, 19, 18)) {
   Serial.println("Falha na inicialização do SD card");
   ESP.restart();
  } else {
    Serial.println("SD card OK");
    delay(100); 
    }

  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);          
  WiFi.setSleep(false);              

  Serial.println("Conectando ao WiFi");
  while (WiFi.status() != WL_CONNECTED) {  
//    Serial.print(".");
      delay(1000);
      semwifi++;
      Serial.println(semwifi);
       if(semwifi > 4) {
          ESP.restart();                               
       }
  }
  
  Serial.print("\nMax Free Heap: ");
  Serial.println(ESP.getMaxAllocHeap());
  Serial.println("");

  myFile = SD.open("atualiza.bin", FILE_WRITE);
   if (myFile) {
     ftp.OpenConnection();
     ftp.ChangeWorkDir("/htdocs");
     String response = "";
     ftp.InitFile("Type I");
     ftp.DownloadString("atualiza.bin", response);
     myFile.println(response);
     delay(100);
     myFile.close();
     } 

     ftp.CloseConnection();
}

void loop() { }

qubits-us · April 5, 2023, 7:28pm

sergioarduino:

 myFile = SD.open("atualiza.bin", FILE_WRITE);
   if (myFile) {
     ftp.OpenConnection();
     ftp.ChangeWorkDir("/htdocs");
     String response = "";
     ftp.InitFile("Type I");
     ftp.DownloadString("atualiza.bin", response);
     myFile.println(response);
     delay(100);
     myFile.close();
     }

change too..

myFile = SD.open("atualiza.bin", FILE_WRITE);
   if (myFile) {
     ftp.OpenConnection();
     ftp.ChangeWorkDir("/htdocs");
     String response = "";
     ftp.InitFile("Type I");
     unsigned char * buf[FIRM_SIZE];
     ftp.DownloadFile("atualiza.bin", buf, FIRM_SIZE);
     myFile.write(buf,FIRM_SIZE);
     delay(100);
     myFile.close();
     } 
#define FIRM_SIZE

completely untested..

sorry ~q

sergioarduino · April 5, 2023, 7:36pm

Oops, tell me my friend. Forgive my lack of knowledge but I put this define inside the setup and also outside it, and it gives this error:

OTAFTP:78:35: error: storage size of 'buf' isn't known
      unsigned char * buf[FIRM_SIZE];
                                   ^
OTAFTP:79:53: error: expected primary-expression before ')' token
      ftp.DownloadFile("atualiza.bin", buf, FIRM_SIZE);
                                                     ^
OTAFTP:80:32: error: expected primary-expression before ')' token
      myFile.write(buf,FIRM_SIZE);
                                ^
exit status 1
storage size of 'buf' isn't known

qubits-us · April 5, 2023, 7:38pm

define at the top of the sketch..

sergioarduino · April 5, 2023, 7:45pm

Looks like it's something with the FTP library. Don't want to compile:

In function 'void setup()':
OTAFTP:77:53: error: no matching function for call to 'ESP32_FTPClient::DownloadFile(const char [13], unsigned char* [20000], int)'
      ftp.DownloadFile("atualiza.bin", buf, FIRM_SIZE);
                                                     ^
In file included from C:\Users\EU\Downloads\OTAFTP\OTAFTP.ino:14:0:
C:\Users\EU\Documents\Arduino\libraries\esp32_ftpclient\src/ESP32_FTPClient.h:57:8: note: candidate: void ESP32_FTPClient::DownloadFile(const char*, unsigned char*, size_t, bool)
   void DownloadFile(const char * filename, unsigned char * buf, size_t length, bool printUART = false);
        ^
C:\Users\EU\Documents\Arduino\libraries\esp32_ftpclient\src/ESP32_FTPClient.h:57:8: note:   no known conversion for argument 2 from 'unsigned char* [20000]' to 'unsigned char*'
OTAFTP:78:32: error: no matching function for call to 'File::write(unsigned char* [20000], int)'
      myFile.write(buf,FIRM_SIZE);
                                ^
In file included from C:\Users\EU\Downloads\OTAFTP\OTAFTP.ino:16:0:
C:\Users\EU\Documents\Arduino\libraries\esp32-micro-sdcard-master/mySD.h:42:18: note: candidate: virtual size_t File::write(uint8_t)
   virtual size_t write(uint8_t);
                  ^
C:\Users\EU\Documents\Arduino\libraries\esp32-micro-sdcard-master/mySD.h:42:18: note:   candidate expects 1 argument, 2 provided
C:\Users\EU\Documents\Arduino\libraries\esp32-micro-sdcard-master/mySD.h:43:18: note: candidate: virtual size_t File::write(const uint8_t*, size_t)
   virtual size_t write(const uint8_t *buf, size_t size);
                  ^
C:\Users\EU\Documents\Arduino\libraries\esp32-micro-sdcard-master/mySD.h:43:18: note:   no known conversion for argument 1 from 'unsigned char* [20000]' to 'const uint8_t* {aka const unsigned char*}'
In file included from C:\Users\EU\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32/Stream.h:26:0,
                 from C:\Users\EU\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32/Arduino.h:147,
                 from C:\Users\EU\Downloads\OTAFTP\OTAFTP.ino:11:
C:\Users\EU\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32/Print.h:70:12: note: candidate: size_t Print::write(const char*, size_t)
     size_t write(const char *buffer, size_t size)
            ^
C:\Users\EU\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32/Print.h:70:12: note:   no known conversion for argument 1 from 'unsigned char* [20000]' to 'const char*'
C:\Users\EU\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32/Print.h:62:12: note: candidate: size_t Print::write(const char*)
     size_t write(const char *str)
            ^
C:\Users\EU\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32/Print.h:62:12: note:   candidate expects 1 argument, 2 provided
exit status 1
no matching function for call to 'ESP32_FTPClient::DownloadFile(const char [13], unsigned char* [20000], int)'

qubits-us · April 5, 2023, 7:48pm

ftp.DownloadFile("atualiza.bin", buf, FIRM_SIZE,false);
or
ftp.DownloadFile("atualiza.bin", buf, sizeof(buf),false);

qubits-us · April 5, 2023, 7:56pm

that's not big enough..
buff needs to be at least size of firmware..

sergioarduino · April 5, 2023, 7:57pm

With two options:

exit status 1
no matching function for call to 'ESP32_FTPClient::DownloadFile(const char [13], unsigned char* [100], int, bool)'

sergioarduino · April 5, 2023, 7:59pm

This is sample library download code:

/******************************************************************************

ESP32-CAM remote image access via FTP. Take pictures with ESP32 and upload it via FTP making it accessible for the outisde network. 
Leonardo Bispo
July - 2019
https://github.com/ldab/ESP32_FTPClient

Distributed as-is; no warranty is given.

******************************************************************************/
#include "Arduino.h"
#include <WiFi.h>
#include <WiFiClient.h> 
#include <ESP32_FTPClient.h>

#define WIFI_SSID ""
#define WIFI_PASS ""

char ftp_server[] = "files.000webhost.com";
char ftp_user[]   = "";
char ftp_pass[]   = "";

ESP32_FTPClient ftp (ftp_server,ftp_user,ftp_pass);

void setup()
{
  Serial.begin( 115200 );

  WiFi.begin( WIFI_SSID, WIFI_PASS );
  
  Serial.println("Connecting Wifi...");
  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }
  Serial.println("");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  Serial.print("\nMax Free Heap: ");
  Serial.println(ESP.getMaxAllocHeap());
  Serial.println("");

  ftp.OpenConnection();

  //Change directory
  ftp.ChangeWorkDir("/public_html/zyro/gallery_gen");

  // Create a new file to use as the download example below:
  ftp.InitFile("Type A");
  ftp.NewFile("helloworld.txt");
  ftp.Write("Hi, I'm a new file");
  ftp.CloseFile();

  //Download the text file or read it
  String response = "";
  ftp.InitFile("Type A");
  ftp.DownloadString("helloworld.txt", response);
  Serial.println("The file content is: " + response);

  // Get the file size
  const char * fileName = "myPhoto.png";
  size_t       fileSize = 0;
  String       list[128];

  // Get the directory content in order to allocate buffer
  // my server response is type=file;size=18;modify=20190731140703;unix.mode=0644;unix.uid=10183013;unix.gid=10183013;unique=809g7c8e92e4; helloworld.txt
  ftp.InitFile("Type A");
  ftp.ContentList("", list);
  for( uint8_t i = 0; i < sizeof(list); i++)
  {
    uint8_t indexSize = 0;
    uint8_t indexMod  = 0;

    if(list[i].length() > 0)
    {
      list[i].toLowerCase();
      
      if( list[i].indexOf(fileName) > -1 )
      {
        indexSize = list[i].indexOf("size") + 5;
        indexMod  = list[i].indexOf("modify") - 1;

        fileSize = list[i].substring(indexSize, indexMod).toInt();
      }

      // Print the directory details
      Serial.println(list[i]);
    }
    else
      break;
  }

  // Print file size
  Serial.println("\nFile size is: " + String(fileSize));

  //Dynammically alocate buffer
  unsigned char * downloaded_file = (unsigned char *) malloc(fileSize);

  // And download the file
  ftp.InitFile("Type I");
  ftp.DownloadFile(fileName, downloaded_file, fileSize, false);

  //Create a new Directory
  ftp.InitFile("Type I");
  ftp.MakeDir("myNewDir");

  //Enter the directory
  ftp.ChangeWorkDir("/public_html/zyro/gallery_gen/myNewDir");

  //And upload the file to the new directory
  ftp.NewFile( fileName );
  ftp.WriteData(downloaded_file, fileSize);
  ftp.CloseFile();

  free(downloaded_file);

  ftp.CloseConnection();
}

void loop()
{

}

qubits-us · April 5, 2023, 8:05pm

that's nicer, uses dynamic buffer..
I also see filename is a char * too..
but they do call DownloadFile..
just quick..

  // Get the file size
  const char * fileName = "atualiza.bin";
  size_t       fileSize = 0;
  String       list[128];
  ftp.OpenConnection();
  ftp.ChangeWorkDir("/htdocs");

  // Get the directory content in order to allocate buffer
  // my server response is type=file;size=18;modify=20190731140703;unix.mode=0644;unix.uid=10183013;unix.gid=10183013;unique=809g7c8e92e4; helloworld.txt
  ftp.InitFile("Type A");
  ftp.ContentList("", list);
  for ( uint8_t i = 0; i < sizeof(list); i++)
  {
    uint8_t indexSize = 0;
    uint8_t indexMod  = 0;

    if (list[i].length() > 0)
    {
      list[i].toLowerCase();

      if ( list[i].indexOf(fileName) > -1 )
      {
        indexSize = list[i].indexOf("size") + 5;
        indexMod  = list[i].indexOf("modify") - 1;

        fileSize = list[i].substring(indexSize, indexMod).toInt();
      }

      // Print the directory details
      Serial.println(list[i]);
    }
    else
      break;
  }

  // Print file size
  Serial.println("\nFile size is: " + String(fileSize));

  if (filesize > 0)
  {
    myFile = SD.open("atualiza.bin", FILE_WRITE);
    if (myFile) {
      ftp.InitFile("Type I");
      //Dynammically alocate buffer
      unsigned char * buf = (unsigned char *) malloc(fileSize);
      ftp.DownloadFile(filename, buf, filesize, false);
      myFile.write(buf, filesize);
      free(buf);
      delay(100);
      myFile.close();
    }
  }

~q

sergioarduino · April 5, 2023, 8:50pm

I changed type I for type A and put the filenames.txt, this file has 24Kb with 1000 names, 1000 lines.

The code copied this file into the SD CARD, but modified the sequence of some lines.

I did this just to test. Because using the update.bin gave GURU MEDITATION right after showing the size of this file on the serial monitor.

In other words, as soon as you tried to download it, it crashed and restarted.

ESP32 wroom, DEV KIT V1, board 1.0.6 arduino IDE 1.8.16.

What a fight !

And without your help I wouldn't even leave the place.

qubits-us · April 5, 2023, 8:54pm

is there enough memory to allocate that buffer??
when the Guru meditates, probably a null pointer..
malloc probably fails..
ota is using a 128 byte buffer reading the file in chunks..
with your ftp lib, i don't think you can work with chunks..
so if you can't allocate the buff, try a different approach..
i thought you had it flying with http or https??

sorry for your troubles..
~q

sergioarduino · April 5, 2023, 9:16pm

I tried until I got tired of http (or https) sites options. Only with yours did it work /nards. Now I try via FTP.

Thank you so much for getting involved again in my questions.

I'll keep trying, even with the almost zero knowledge I have.

Hugs. But if you have any ideas...

sergioarduino · April 6, 2023, 11:58am

I found another FTP library and decided to test it. See:

//https://github.com/pschatzmann/TinyFTPClient

#include "Arduino.h"
#include <WiFi.h>
#include "ArduinoFTPClient.h"

FTPClient client;

void setup() {
    Serial.begin(115200);

    // connect to WIFI
    WiFi.begin("zzzzzzzzzzzzzzzzzz", "zzzzzzzzzzzzzzzzzzzzz");
    while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
    }

    IPAddress localAddress = WiFi.localIP();
    Serial.println();
    Serial.print("Started with address ");
    Serial.println(localAddress.toString());

    // optional logging
    FTPLogger::setOutput(Serial);
    FTPLogger::setLogLevel(LOG_DEBUG);
    
    // open connection
    client.begin(IPAddress(185,27,134,11), "ttnh_33936483", "zzzzzzzzzzzzzzz");

    // copy data from file
    FTPFile file = client.open("/htdocs/atualiza.bin");
    int len;
    byte buffer[100];
    while (file.available()>0){
        len = file.read(buffer,100);
        Serial.write(buffer,len);
    }

    // cleanup
    file.close();
    client.end();
}


void loop() {
}

I want to download the update.bin file that is in the htdocs folder on the server. Using fileZilla works perfectly.

But unfortunately I see these error messages. Does anyone know what can it be ?

FTP INFO - FTPClient begin
FTP DEBUG - FTPBasicAPI  open
FTP DEBUG - FTPBasicAPI::connect connect 185.27.134.11:21
FTP DEBUG - FTPBasicAPI::checkResult 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------

FTP DEBUG - FTPBasicAPI::checkResult - checking with 220
FTP DEBUG - FTPBasicAPI::checkResult  -> success with 220
FTP DEBUG - FTPBasicAPI::cmd USER ttnh_33936483
FTP DEBUG - FTPBasicAPI::checkResult 220-You are user number 1305 of 6900 allowed.

FTP DEBUG - FTPBasicAPI::checkResult - checking with 331
FTP DEBUG - FTPBasicAPI::checkResult - checking with 230
FTP DEBUG - FTPBasicAPI::checkResult - checking with 530
FTP ERROR - FTPBasicAPI::checkResult USER ttnh_33936483
FTP ERROR - FTPBasicAPI::checkResult 220-You are user number 1305 of 6900 allowed.

FTP INFO - FTPClient open
FTP DEBUG - FTPFile
FTP DEBUG - FTPBasicAPI read
FTP DEBUG - FTPBasicAPI::cmd RETR /htdocs/atualiza.bin
FTP DEBUG - FTPBasicAPI::checkResult 220-Local time is now 07:13. Server port: 21.

FTP DEBUG - FTPBasicAPI::checkResult - checking with 150
FTP DEBUG - FTPBasicAPI::checkResult - checking with 125
FTP ERROR - FTPBasicAPI::checkResult RETR /htdocs/atualiza.bin
FTP ERROR - FTPBasicAPI::checkResult 220-Local time is now 07:13. Server port: 21.

FTP DEBUG - FTPBasicAPI checkClosed -> client is closed
FTP DEBUG - FTPFile available: 0
FTP INFO - FTPFile close
FTP DEBUG - FTPBasicAPI::checkResult 220-This is a private system - No anonymous login

FTP DEBUG - FTPBasicAPI::checkResult - checking with 226
FTP ERROR - FTPBasicAPI::checkResult close
FTP ERROR - FTPBasicAPI::checkResult 220-This is a private system - No anonymous login

FTP DEBUG - FTPBasicAPI closeData
FTP INFO - FTPClient end
FTP DEBUG - FTPBasicAPI quit
FTP DEBUG - FTPBasicAPI::cmd QUIT
FTP DEBUG - FTPBasicAPI::checkResult 220 You will be disconnected after 60 seconds of inactivity.

FTP DEBUG - FTPBasicAPI::checkResult - checking with 221
FTP DEBUG - FTPBasicAPI::checkResult - checking with 226
FTP ERROR - FTPBasicAPI::checkResult QUIT
FTP ERROR - FTPBasicAPI::checkResult 220 You will be disconnected after 60 seconds of inactivity.

I'm using a wroom ESP32 alone, only with USB cable on PC, Windows 10, arduino IDE 1.8.16 board ESP32 1.0.6

Topic		Replies	Views
Duplicating a .bin file Storage	6	1387	May 6, 2021
Eps32 WiFiClientSecure download file Programming Questions	4	639	January 23, 2024
ESP32 SD-Card Stream download, sd card failure? Programming Questions	3	1641	August 6, 2022
ESP32 SIM800L File.Available() & File.Read() Issue - Inverted question mark ⸮ Programming Questions	13	2260	May 6, 2021
How to download logged *.txt file from (ESP32 + SD Card) connected to home WiFi..? Project Guidance	8	7276	October 15, 2021

Downloading .bin from FTP and saving to SD CARD

Related topics