[SOLVED] Loop reading on SD card and sending HTTP requests GET

Good evening,
From a Mega and an Ethernet / SD shield, I try to send on my local website, via client.print, a series of GET requests including the data of a datalogger read through a while loop and openNext of Sdfat. One query per file on the SD card.
The problem is that I do not know how to delay the sending of GET requests because my website throws me a RST after a dozen requests. If I introduce any delay () in the loop, I get thrown after 2 requests.

The contents of a file:

GET /emoncms/input/bulk?data=[[34,4,{"V":25870},{"I":15},{"VPV":48},{"PPV":65},{"CS":95},{"ERR":99},{"H19":32},{"H20":147},{"H21":258},{"H22":963},{"H23":5789},{"HSDS":11}],[36,5,{"MODE":25870},{"CS":15},{"AC_OUT_V":48},{"AC_OUT_I":65},{"V":95},{"AR":99},{"WARN":75}]]&time=1539036950&apikey=aaaaaaaaaaaaabbbbbbbbbbbbbbcccccccccccc
HTTP/1.1
Host:localhost
User-Agent: Arduino-ethernet
Connection: close

and the reading code of the SD card:

void readSD() {

  File logFile;
  char fileName;
  sd.ls();

  sd.vwd()->rewind();

  while (logFile.openNext(sd.vwd(), O_READ | O_WRITE)) {
    String dataBuff = logFile.readStringUntil('\0');
    client.print(dataBuff);
    //delay(10000);
    Serial.println(dataBuff);
    if (!logFile.remove()) Serial.println(F("Error logFile.remove"));
    }

  if (!sd.exists(fileName)) {
    Serial.println(F("Nothing to send from SD card"));
  }
}

The server connection code :

void doHTTP() {

  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

  if (!client.connected() && (lastConnected && (millis() - lastConnectionTime >= (postingInterval + 5000)))) {
    Serial.println(F("\r\n"));
    Serial.println(F("Disconnecting..."));
    client.stop();
  }

  if (!client.connected() && (millis() - lastConnectionTime > postingInterval)) {
    if (client.connect(server, 80)) {
      Serial.print(F("\r\n"));
      Serial.println(F("Connected"));
      readSD();
      sendData();
      lastConnectionTime = millis();
    }

    // if the server's disconnected more than 2 min, stop the client and write data on sd card:
    if (!client.connected() && (millis() - (lastConnectionTime >=  (postingInterval * 12)))) {
      Serial.println(F("disconnecting from server."));
      client.stop();
      writeSD();
      //delay(30000);
      //delay(1000);
    }
  }
  lastConnected = client.connected();
}

These are code excerpts. Post complete code, don't waste our time!

The problem is that I do not know how to delay the sending of GET requests because my website throws me a RST after a dozen requests.

What is your "local website" in this context? The Arduino? What exactly does "throwing a RST" mean? Does it end the TCP connection?

Sorry for my bad presentation. Perhaps my english isn't really good.
My local web site is on my PC. I don't know if it possible to have a web server and a web client in the same time on Arduino.
The RST answer of my web site is at the end of the exchange between ethernet shield and my web server. And it interrupts the connection.

#include <Arduino.h>
#include "SdFat.h"
#include <SPI.h>
#include <RTClib.h>
#include <Wire.h>
#include <Ethernet.h>

RTC_DS1307 rtc;
SdFat sd;
File logFile;
#define FILE_BASE_NAME "Data"

const uint8_t chipSelect = 4;
const char fileName[] = "datalog.txt";

// Error messages stored in flash.
#define error(msg) sd.errorHalt(F(msg))


// this must be unique
byte mac[] = { 0x90, 0xA2, 0xDA, 0x00, 0x59, 0x67 };

// change to your network settings
byte ip[] = { 192, 168, 5, 53 };
byte gateway[] = { 192, 168, 5, 1 };
byte subnet[] = { 255, 255, 255, 0 };

char server[] = "192.168.5.50";

EthernetClient client;


String apikey = "aaaabbbbbcccccdddddddeeeeeeeee";  //localhost api key

unsigned long lastConnectionTime = 0;          // last time you connected to the server, in milliseconds
boolean lastConnected = false;                 // state of the connection last time through the main loop
const unsigned long postingInterval = 10 * 1000; // delay between updates, in milliseconds

//============================================================

int value1[17] = {25341, 440, 5879, 222478, 15, 48, 65, 95, 99, 75, 32, 147, 258, 963, 5789, 11, 28};
int value2[11] = {25341, 440, 5879, 222478, 15, 48, 65, 95, 99, 75, 32};

//=============================================================
void dateTime(uint16_t* date, uint16_t* time) {

  DateTime now = rtc.now();

  // return date using FAT_DATE macro to format fields
  *date = FAT_DATE(now.year(), now.month(), now.day());

  // return time using FAT_TIME macro to format fields
  *time = FAT_TIME(now.hour(), now.minute(), now.second());
}

//============================================================
void writeSD() {
	const uint8_t BASE_NAME_SIZE = sizeof(FILE_BASE_NAME) - 1;
	char fileName[13] = FILE_BASE_NAME "00.csv";

	// Find an unused file name.
	if (BASE_NAME_SIZE > 6) {
		("FILE_BASE_NAME too long");
	}
	while (sd.exists(fileName)) {
		if (fileName[BASE_NAME_SIZE + 1] != '9') {
		fileName[BASE_NAME_SIZE + 1]++;
	} else if (fileName[BASE_NAME_SIZE] != '9') {
		fileName[BASE_NAME_SIZE + 1] = '0';
     		fileName[BASE_NAME_SIZE]++;
	} else {
		error("Can't create file name");
	}
}

  DateTime now = rtc.now();
  unsigned long currentTime = now.unixtime();

  SdFile::dateTimeCallback(dateTime);
  if (!logFile.open(fileName, O_CREAT | O_WRITE | O_EXCL)) {
  	error("write_logFile.open");
  }

  logFile.print("GET /emoncms/input/bulk?data=[[");
  /*logFile.print("34,0,{\"temp\":");
    logFile.print(temperature);
    logFile.print("}],[34,1,{\"H2\":");
    logFile.print(MQGetGasPercentage(MQRead(MQ_PIN\) / Ro, GAS_H2));
    logFile.print("}],[34,2,{\"power1\":");
    logFile.print(Irms1 * volt);
    logFile.print("}],[34,3,{\"power2\":");
    logFile.print(Irms2 * volt);
    logFile.print("}],[");*/
  logFile.print("34,4,{\"V\":");
  logFile.print(value1[3]);
  logFile.print("},{\"I\":");
  logFile.print(value1[4]);
  logFile.print("},{\"VPV\":");
  logFile.print(value1[5]);
  logFile.print("},{\"PPV\":");
  logFile.print(value1[6]);
  logFile.print("},{\"CS\":");
  logFile.print(value1[7]);
  logFile.print("},{\"ERR\":");
  logFile.print(value1[8]);
  logFile.print("},{\"H19\":");
  logFile.print(value1[10]);
  logFile.print("},{\"H20\":");
  logFile.print(value1[11]);
  logFile.print("},{\"H21\":");
  logFile.print(value1[12]);
  logFile.print("},{\"H22\":");
  logFile.print(value1[13]);
  logFile.print("},{\"H23\":");
  logFile.print(value1[14]);
  logFile.print("},{\"HSDS\":");
  logFile.print(value1[15]);
  logFile.print("}],[36,5,{\"MODE\":");
  logFile.print(value2[3]);
  logFile.print("},{\"CS\":");
  logFile.print(value2[4]);
  logFile.print("},{\"AC_OUT_V\":");
  logFile.print(value2[5]);
  logFile.print("},{\"AC_OUT_I\":");
  logFile.print(value2[6]);
  logFile.print("},{\"AR\":");
  logFile.print(value2[7]);
  logFile.print("},{\"WARN\":");
  logFile.print(value2[8]);
  /*logFile.print("}],[34,6,{\"V\":");
    logFile.print(value3[1]);
    logFile.print("},{\"I\":");
    logFile.print(value3[2]);
    logFile.print("},{\"P\":");
    logFile.print(value3[3]);
    logFile.print("}, {\"CE\":");
    logFile.print(value3[4]);
    logFile.print("},{\"SOC\":");
    logFile.print(value3[5]);
    logFile.print("},{\"TTG\":");
    logFile.print(value3[6]);
    logFile.print("},{\"AR\":");
    logFile.print(value3[9]);
    logFile.print("},{\"BMV\":");
    logFile.print(value3[10]);
    logFile.print("},{\"H1\":");
    logFile.print(value3[13]);
    logFile.print("},{\"H2\":");
    logFile.print(value3[14]);
    logFile.print("},{\"H3\":");
    logFile.print(value3[15]);
    logFile.print("},{\"H4\":");
    logFile.print(value3[16]);
    logFile.print("},{\"H5\":");
    logFile.print(value3[17]);
    logFile.print("},{\"H6\":");
    logFile.print(value3[18]);
    logFile.print("},{\"H7\":");
    logFile.print(value3[19]);
    logFile.print("},{\"H8\":");
    logFile.print(value3[20]);
    logFile.print("},{\"H9\":");
    logFile.print(value3[21]);
    logFile.print("},{\"H10\":");
    logFile.print(value3[22]);
    logFile.print("},{\"H11\":");
    logFile.print(value3[23]);
    logFile.print("},{\"H12\":");
    logFile.print(value3[24]);
    logFile.print("},{\"H17\":");
    logFile.print(value3[25]);
    logFile.print("},{\"H18\":");
    logFile.print(value3[26]); */
  logFile.print("}]]&time=");
  logFile.print(currentTime - 7200);
  logFile.print("&apikey=");
  logFile.println(apikey);
  logFile.println("HTTP/1.1");
  logFile.println("Host:localhost");
  logFile.println("User-Agent: Arduino-ethernet");
  logFile.println("Connection: close");
  logFile.println();
  // Force data to SD and update the directory entry to avoid data loss.
  if (!logFile.sync() || logFile.getWriteError()) {
    error("logFile write error");
  }
  logFile.close();
  Serial.println(F("Data writed"));
}
//============================================================
void readSD() {

  File logFile;
  char fileName;
  sd.ls();

  sd.vwd()->rewind();
  while (logFile.openNext(sd.vwd(), O_READ | O_WRITE)) {
  String dataBuff = logFile.readStringUntil('\0');
  client.print(dataBuff);
  //delay(10000);
  Serial.println(dataBuff);
  if (!logFile.remove()) Serial.println(F("Error logFile.remove"));
  }
  if (!sd.exists(fileName)) {
  Serial.println(F("Nothing to send from SD card"));
  }
}

//============================================================
void sendData() {

  DateTime now = rtc.now();
  unsigned long currentTime = now.unixtime();
  //Serial.println(currentTime);

  client.print("GET /emoncms/input/bulk?data=[[");
  client.print("32,4"); // node MPPT-150_35
  client.print(",{\"V\":");
  client.print(value1[3]);
  client.print("},{\"I\":");
  client.print(value1[4]);
  client.print("},{\"VPV\":");
  client.print(value1[5]);
  client.print("},{\"PPV\":");
  client.print(value1[6]);
  client.print("},{\"CS\":");
  client.print(value1[7]);
  client.print("},{\"ERR\":");
  client.print(value1[8]);
  client.print("},{\"H19\":");
  client.print(value1[10]);
  client.print("},{\"H20\":");
  client.print(value1[11]);
  client.print("},{\"H21\":");
  client.print(value1[12]);
  client.print("},{\"H22\":");
  client.print(value1[13]);
  client.print("},{\"H23\":");
  client.print(value1[14]);
  client.print("},{\"HSDS\":");
  client.print(value1[15]);
  client.print("}],[");
  client.print("34,5"); // node Inverter_24V_500VA
  client.print(",{\"MODE\":");
  client.print(value2[3]);
  client.print("},{\"CS\":");
  client.print(value2[4]);
  client.print("},{\"AC_OUT_V\":");
  client.print(value2[5]);
  client.print("},{\"AC_OUT_I\":");
  client.print(value2[6]);
  client.print("},{\"V\":");
  client.print(value2[7]);
  client.print("},{\"AR\":");
  client.print(value2[8]);
  client.print("},{\"WARN\":");
  client.print(value2[9]);
  client.print("}]]&time=");
  client.print(currentTime - 7200);
  client.print("&apikey=");
  client.print(apikey);
  client.println(" HTTP/1.1");
  //client.println("Host:genatfab.ovh");
  client.println("Host:localhost");
  client.println("User-Agent: Arduino-ethernet");
  client.println("Connection: close");
  client.println();
}

//============================================================
void doHTTP() {

  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

  if (!client.connected() && (lastConnected && (millis() - lastConnectionTime >= (postingInterval + 5000)))) {
    Serial.println(F("\r\n"));
    Serial.println(F("Disconnecting..."));
    client.stop();
  }

  if (!client.connected() && (millis() - lastConnectionTime > postingInterval)) {
    if (client.connect(server, 80)) {
      Serial.println(F("\r\n"));
      Serial.println(F("Connected"));
      readSD();
      sendData();
      lastConnectionTime = millis();
    }

    // if the server's disconnected more than 2 min, stop the client and write data on sd card:
    if (!client.connected() && (millis() - (lastConnectionTime >=  (postingInterval * 12)))) {
      Serial.print(F("\r\n"));
      Serial.println(F("disconnecting from server."));
      client.stop();
      writeSD();
      Serial.println(F("Writing on SD card"));
      Serial.println(F("\r\n"));
      delay(30000);
    }
  }
  lastConnected = client.connected();
}
//============================================================
void setup() {
  Serial.begin(115200);

  // Mega
  pinMode(53, OUTPUT);
  // disable w5100 SPI while setting up SD
  pinMode(10, OUTPUT);
  digitalWrite(10, HIGH);
  // enable SD SPI
  pinMode(4, OUTPUT);
  digitalWrite(4, LOW);

  rtc.begin();

  if (! rtc.isrunning()) {
    Serial.println("RTC ne fonctionne PAS!");
    // La ligne qui suit ajuste le RTC à la date et time du moment de compilation
    rtc.adjust(DateTime(__DATE__, __TIME__));
  }

  if (!sd.begin(chipSelect, SD_SCK_MHZ(50)))
  {
    sd.initErrorHalt();
  }

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  digitalWrite(10, HIGH);

  Serial.println(Ethernet.localIP());

  //readSD();
  //writeSD();

}
//============================================================
void loop() {
  doHTTP();
}

for more precisions, it seems to me that client.print sends the requests too quickly to the web server. That's why I think I need to add a delay () somewhere in the code

    if (!client.connected() && (millis() - (lastConnectionTime >=  (postingInterval * 12)))) {

This line is probably not doing what you intended. Hint: check the braces!

Thanks for the remark. But does this explain why client.print sends no less than 65 queries one after the other in less than a few seconds without any "HTTP 200 OK" until you get a RST from the web server and have an interrupt of the connexion ?
In attachment, the screenshot of wireshark.

But does this explain why client.print sends no less than 65 queries one after the other in less than a few seconds without any "HTTP 200 OK" until you get a RST from the web server and have an interrupt of the connexion ?

Yes, it does.

I corrected the line concerned as I think it's allright: if (!client.connected() && (millis() - lastConnectionTime >=  postingInterval * 12)) {The requests are send at the frequency of nearly one per seconde but whatever i do, no more 30 GET requests are accepted by the server and I have the same final answer : RST and interruption of the connection. That is problematic because I can have 9,999 files stored on the sd card

I'm still waiting for complete code (which I can compile in my IDE).

I put it to #2 et #3 :slight_smile:

I put it now in an attachment

Emoncms_SD_test_boucle.ino (10.2 KB)

Change

  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

to

  while (client.available()) {
    char c = client.read();
    Serial.write(c);
  }

to read the server response completely.

The requests are send at the frequency of nearly one per seconde but whatever i do, no more 30 GET requests are accepted by the server and I have the same final answer : RST and interruption of the connection. That is problematic because I can have 9,999 files stored on the sd card

Read the server response after every request and close the connection as you tell the server in the header.

There is no more server response message display in the serial console with "while ... write" than with "if... print".

I will try to send requests on an another http (because https don't work on Arduino from what I understood) web site to see if, in another way, it is not a problem of security configuration of my server. The request :

GET / HTTP/1.1
Host: perdu.com
User-Agent: arduino-ethernet
Connection: close

There is only one response from the server that appears at the very end after the requests, regardless of the number of requests. If I put a delay before

client.print(dataBuff);

, this delay is take on board just before the server response

The serial console display :

disconnecting from server.
Writing Data0004 on SD card



Connecting ...
Data0000
Data0001
Data0002
Data0003
Data0004
GET / HTTP/1.1
Host: perdu.com
User-Agent: arduino-ethernet
Connection: close


GET / HTTP/1.1
Host: perdu.com
User-Agent: arduino-ethernet
Connection: close


GET / HTTP/1.1
Host: perdu.com
User-Agent: arduino-ethernet
Connection: close


GET / HTTP/1.1
Host: perdu.com
User-Agent: arduino-ethernet
Connection: close


GET / HTTP/1.1
Host: perdu.com
User-Agent: arduino-ethernet
Connection: close


Nothing to send from SD card
HTTP/1.1 200 OK
Date: Fri, 12 Oct 2018 21:31:55 GMT
Server: Apache
Last-Modified: Thu, 02 Jun 2016 06:01:08 GMT
ETag: "cc-5344555136fe9"
Accept-Ranges: bytes
Content-Length: 204
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

<html><head><title>Vous Etes Perdu ?</title></head><body><h1>Perdu sur l'Internet ?</h1><h2>Pas de panique, on va vous aider</h2><strong><pre>    * <----- vous &ecirc;tes ici</pre></strong></body></html>

In fact, the only one "HTTP 200 OK" response is sent by the server after the first request, but appears at the end in the serial console. I could actually verify it with wireshark

I seem to be approaching a resolution (or not ...?):
I've changed ("Connection: close"); to ("Connection: keep-alive");
With wireshark, I can see this :
for about 110 requests, the first ten requests have their "HTTP 200 OK" response per request. From the 11th, no response, but the receiver (my web server) announce a window of 24880 bytes ("TCP window full" on wireshark), then an ACK response from Arduino following by a sequence "TCP zero window" from arduino for the GET request, then an ACK response from server, and so on. Track all subsequent the "HTTP 200 OK" responses in a row for the 100 remaining.

For more than 115 or 120 requests , "HTTP 200 OK" response until the 10th. Then a succession of "TCP windwo full", "TCP window zero", ACK like previous, but at the end there is three RST responses and an interruption of the connection. And none "HTT 200 OK ".

Why my server send this reset in the middle of a valid TCP connection?

In attachment, wireshark capture files with txt extension but these are pcap files. First file with successful 111 requests. Second file with unsuccessful 149 requests. For better visibility, the files must be filtered with the address 192.168.5.53

with_HTTP_OK_without_RST.txt (756 KB)

with_RST.txt (112 KB)

I found ! Thanks Pylon for your help !
Searching on the side of openenergymonitor.org in this sketch NanodeRF_emoncms_dataPull.ino :

https://github.com/openenergymonitor/emonRemote/blob/master/NanodeRF_emoncms_dataPull/NanodeRF_emoncms_dataPull.ino

As my server only returns "HTTP 200 OK" responses for each of the first 10 requests, I added an increment in the read loop of the SD card. and now I can send up to 9999 requests in groups of up to 10 without the server terminating the connection, without RST sequence :

void readSD() {

  File logFile;
  char fileName;
  sd.ls();
  int i = 0; 
  sd.vwd()->rewind();
  while (logFile.openNext(sd.vwd(), O_READ | O_WRITE) && i<10) {
    String dataBuff = logFile.readStringUntil('\0');
    client.print(dataBuff);
     Serial.println(dataBuff);
    if (!logFile.remove()) Serial.println(F("Error logFile.remove"));
    //delay(500);
     i++;
  }

  if (!sd.exists(fileName)) {
    Serial.println(F("Nothing to send from SD card"));
    return 0;
  }
}

Emoncms_SD_test_boucle.ino (10.3 KB)