Improve WIFI data transfer speed

Thanks in advance for your attention

I am in a proyect where I read data stored in a microSD.
The data is saved in a document line by line separating values with commas up to 6 different sensors value data.
Each line is printed in the microSD with a sample rate of 100Hz.
After saving all this information in the microSD, I read it line by line and send each line by wifi to a sever in a Raspberry Pi.
However for 1 minute of recording I have 2 minutes aproximately to send all this data.
I already tried to store 10 lines of this data in a buffer and send this buffer every 10 lines however I have only reduced the transfer time in just 1 minute.
Can I improve my data transfer speed by WIFI? How?

Here is my code:

void readSD(){
  //------------------------------------------------------------------------------
  // OPEN SD FILE AND REED VALUES TO SEND BY WIFI
  //------------------------------------------------------------------------------
  file = SD.open("File.txt",FILE_READ);  
  if (file) {
    while (file.available()) {
      char charc = file.read();
      if(charc=='\r'){
        while (file.read() != '\n');               // Descartar el resto hasta llegar a la siguiente línea
        cont=cont+1;;
        if (cont==10){
          sendDataWIFI();
          data = ""; 
          cont=0;
        }         
      } else {
        data = data + charc;
      }
    }
    sendDataWIFI();
    data = "";
    // close the file:
    archivo.close();
  } else {
    // if the file didn't open, print an error:    
    if (mySerial == true){
      Serial.println("Error trying to open File.txt");
    }
  }
void sendDataWIFI(){
  if (client.connect(server, 80)>0) {                 // Connect with server
    client.print("GET /recieverdata.php?data=");
    client.print(data);
    client.println(" HTTP/1.0");
    client.println("User-Agent: Arduino 1.0");
    client.println();
    if (mySerial == true){
      Serial.println("Conectado");
    }
  } else {
    if (mySerial == true){
      Serial.println("Fail in the connection");
    }        
  }
  if (!client.connected()) {
    if (mySerial == true){
      Serial.println("Disconnected!");
    }
  }
  client.stop();
  client.flush();
}

void setupWIFI() {
  // check for the WiFi module:
  if (WiFi.status() == WL_NO_MODULE) {
    if (mySerial == true){
      Serial.println("Communication with WiFi module failed!");
    }
    // don't continue
    while (true);
  }

  String fv = WiFi.firmwareVersion();
  if (fv < WIFI_FIRMWARE_LATEST_VERSION) {
    if (mySerial == true){
      Serial.println("Please upgrade the firmware");
    }    
  }

  // attempt to connect to Wifi network:
  while (status != WL_CONNECTED) {
    if (mySerial == true){
      Serial.print("Attempting to connect to SSID: ");
      Serial.println(ssid);
    }
    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);

    // wait 10 seconds for connection:
    delay(10000);
  }
  if (mySerial == true){
    Serial.println("Connected to wifi");
  }
  printWifiStatus();

  Serial.println("\nStarting connection to server...");
  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    if (mySerial == true){
      Serial.println("connected to server");
    }
  }
}

Also in the server I store all this data in another .txt file with this code:

<?php
		$data1 = $_GET['data'];
		$direc='File.txt';
		$file1 = fopen($direc, 'a') or die("Error trying to open the file");
		fwrite($file1, $data1 . PHP_EOL);
		fclose($file1);
?>

I setup a system like yours but with a few modifications:

  • no SD card, fixed String data + a counter (~40 data characters per line)
  • Arduino connected to laptop Windows 10 Mobile Hotspot to route data trough laptop
  • laptop connected to WiFi router
  • Raspberry Pi 4 Apache Server with PHP connect to router trough Ethernet
  • your PHP script converted to English only :slight_smile:
  • Windows 10 Task Manager show network performance for Hotspot
  • sending data while pushing a button to ensure sustained transfer for measurement

I received about 60 lines of data per second (similar to your setup). During that time, the Task Manager showed a received data rate of ~250kBit/s. That means the 24kBytes per second resulted in a raw data transfer rate of 2.4kBytes per second (10%).

I confirmed the data by looking at the network traffic with WireShark and the data file.

WireSharks shows each request is sent in a separate package, there are multiple ACK TCP packages send in between, and the server sends HTTP OK packets back.

Your approach seems to be fundamentally wasting a lot of data rate by sending unnecessary data e.g. GET, recieverdata.php, HTTP/1.0, User-Agent ...

It looks to me like you need another protocol. From what I guess you are basically calling web page links and the server creates a log file of the links you called.

I do not have a solution right now, but I already wanted to give you something to think about and I have a few questions.

  • Is there a reason why you chose the HTTP protocol?
  • Did you investigate other options inside HTTP?
  • Did you consider MQTT, FTP or other protocols?

I will give this some thought and get back to you. Maybe somebody else has a pointer to get you started as well.

Thaks for all your comments Klaus_K.

I need my device to have an SD card because I save data in it for 30 minutes with a sample rate of 100Hz and also I need It to be wireless and portable.

The server is implemented in a Raspberry Pi 4 model B with LAMP web server.

The reason I choosed HTTP is because I am learning and I did not know anything at all of other protocoles, also I found really easy to use HTTP and GET.

I don´t have any knowledge in other protocols so I will get deep into it.

Anyway if you have some suggestions in which protocol will suit better in my project I will kindly appreciate.

Thanks

Understood, I just wanted to eliminate the SD card as a limiting source.

I did a little bit of testing and found that when you use the Arduino as server you can download data at a much higher rate using HTTP.

This might not always be possible e.g., when you use a public cloud server. Then having the Arduino as client is safer.
In your case (if both devices are in your network) the Raspberry Pi can act as client and download files from the Arduino. There likely are a lot of ways to write a small HTTP client on a Raspberry Pi. You can just use a web browser to do a manual save-file-as download.

Additionally, I believe downloading files is likely safer than uploading files. I cannot give you an expert argument for that. Just from a few things I read, it sounds like most issues come from server accepting data (because it can be manipulated to create faulty states) and less from providing data.