I am working on a field unit for sensor data collection. Currently I'm sampling the sensors and writing the data to an SD-card, but the project lead (non-technical) has requested that the data be uploaded to a remote server as well.
IMPORTANT NOTES
Hardware
Arduino MEGA 2560
Mayhew Enhanced ADC shield (analog to digital sensor module)
Adafruit Datalogging shield (for data storage on SD card)
Botletics LTE/GPS shield (for GPS and date/time data as well as connectivity for upload)
Options found and concerns
1. Upload via FTP
The data needs to be uploaded to a secure site, and at this time the client will not allow connections via FTP. It may be possible to implement with certificates, however this would be one of (if not) the last options.
2. Upload via POST/GET
This would theoretically be possible if I can upload the data as recorded, however this is a +- 100 character string (char*) of values that gets collected every second. The units will be deployed in remote sites (on mountains, in deserts, etc.) and power consumption is a problem. We cannot have the cellular module running 24/7. We need to collect the data and transmit once per day.
3. Attached Pi and upload via SSH
Same problem as number 2. We cannot provide power to maintain a Pi for extended periods.
I've seen some forum posts where people refer to streaming data, which is something I don't know much about. Each file will be about 8.7MB.
Does anyone know of other methods of transferring data from an Arduino via cellular module to a remote server?
That question depends on what you want to do on the server side and which protocols are supported by a library.
If you want to save the files, then FTP is probably a good choice. You could also check what kind of protocols services like Dropbox use.
Just because you store the data in a file does not mean you need to send a file. You could use a protocol like MQTT to send the data as individual sensor readings. The MQTT protocol has been designed as lightweight sensor data protocol. In many cases the data is send as it becomes available to avoid storing the data on the sensor. Because your radio seems to require a lot of setup time and power it makes sense to send the data in bulk. So, every now and then you can send as many messages as you have data sets. MQTT is supported by many platforms from Arduino, Raspberry Pi to Cloud solutions. It is based on TCP/IP. This would allow you to find an easy trade-off between power availability (battery size, solar panel, ...) and real-time data display.
With regards to the files... yes, unfortunately they want to store them in csv/txt files specifically, as these files will need to be accessed by date, by research institutions. On the server side I can do pretty much whatever I want, but as in all companies, getting into the server (security) is the challenge.
Thank you for your ideas relating to MQTT, I have not heard of it before and will do some research pertaining thereto.