Uploading large files (between 5 and 15MB) with FTP using Mega2560 and SIM7000

Projects Networking, Protocols, and Devices
1

I have a field unit collecting and saving data on a datalogging shield (SD-card). Every 24-hours I need to upload this data to a remote server. The saved file is usually about 8MB, however the scientists are still finalizing their requirements (the number of samples they want to take) and the data could increase.

I have an FTP server configured and running to upload the files to, the FTP server is working.

RESTRICTION

The device cannot remain on, uploading the data as it samples, because the device has a limited battery.

IMPORTANT NOTES

Hardware

Arduino MEGA 2560
Mayhew Enhanced ADC shield (analog to digital sensor module)
Adafruit Datalogging shield (for data storage on SD card)
Botletics SIM7000 LTE/GPS shield (for GPS and date/time data as well as connectivity for upload)

I've been looking into sending the files using FTPEXTPUT, however the AT command manual states the data length has to be between 0 and 300k. The examples I've found also read the entire file into memory before transmission (which I obviously can't do because of the memory constraints on the Mega).

I have very little experience with electronics, no experience at all with AT components and am learning for this project.

Has anyone transferred large files like these using Arduino's before? I would appreciate any help, even just a point to some additional resources.

Thank you.

2

There's a page in the playground here which has the code for sending or receiving data to SD card from FTP.

It buffers the data it's sending so it should be able to handle files of arbitrary length.

3

Hi wildbill,

Thank you for your suggestion.

I had previously looked at the playground page. The main difference is that I'm connecting via Cellular connection (LTE) and not ethernet. As such (unless I am misunderstanding the process) I have to use AT commands to communicate with, and transfer the data using the SIM7000 module.

From what I've researched it seems like the

AT+FTPEXTPUT

command is used to transfer large files using the SIM7000 module, however the documentation of the module shows that the data length has to be between 0 and 300k.

Here is a screenshot of that section in the manual, and

Here (SIM7000 Series_AT Command Manual_V1.04.pdf) is a link to the entire manual for your perusal.

It is very possible that I'm misinterpreting the implementation, however if that is the case I would very much appreciate if you wouldn't mind helping me to understand.

Thank you.