Can't stop read of multiple copies of HTML form POST data

I just don't understand what firefox is doing here!

I can successfully upload a web page stored on an SD card attached to my arduino.

I can successfully post the form and receive the HTML POST header and then view it in serial monitor.

I can successfully extract the POST data length from the header.

But I cannot stop my loop for reading multiple copies of the same POST data - I only want one copy of it. Any further copies, due to firefox sending multiple requests, can be discarded.

Either the POST data length in the HTTP header is incorrect, my loop is faulty some how or SD library files don't behave the same way as I am sued to with C++ on other platforms.

with the SD library, when you open a file with CREAT and WRITE attributes, are the file contents erased as expected or are all subsequent writes appended to the end?

The two functions involved are below and the relevant full source files area attached.

NOTE: I have removed commented out code below for clarity, and it remains commented out in my source file until I get the basic read loop right.

Basically the loop dumps out the POST data to a file because I need to do processing on it, it can potentially be quite long and the processing is to slow and holds up my HTTP response to Firefox.
So I intend to do the POST data processing from the file after I send the HTTP response.

Also below is the contents of the file containing the POST data after one POST event in firefox.
It should contain only one copy of 'Station1' to 'Station8', however it contains many copies.

bool CWifi::readPostData(WiFiEspClient& WifiClient, const uint16_t nPostDataLength)
{
  bool bEndOfDataFound = true;
  String strTemp, strFileName = F("post.txt");
  CTextFile filePostData(strFileName.c_str(), O_WRITE | O_CREAT);
  CTextFile fileProgram(m_strIrrigProgFileName.c_str(), /*O_WRITE | O_CREAT*/FILE_READ);
  uint16_t nI = 0;
  
  if (fileProgram && filePostData)
  {
    m_nBytesRead = 0;
    memset(m_arrayReadBuff, 0, BUFF_SIZE);
    nI = 0;
    while ((WifiClient.available() > 0) && (m_nBytesRead < nPostDataLength))
    {
      m_arrayReadBuff[nI++] = WifiClient.read();
      m_nBytesRead++;
      if (nI >= BUFF_SIZE)
      {
        filePostData.write(m_arrayReadBuff, BUFF_SIZE);
        memset(m_arrayReadBuff, 0, BUFF_SIZE);
        nI = 0;
      }
    }
    if (nI > 0)
      filePostData.write(m_arrayReadBuff, nI);
    filePostData.close();

  }
  return bEndOfDataFound;

}

uint16_t CWifi::getPostDataLength(String& strHTTPReq)
{
  // EXAMPLE
  // =======
  // HTTP request:
  // POST /program.htm HTTP/1.1
  // Host: 10.0.0.79
  // User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
  // Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  // Accept-Language: en-US,en;q=0.5
  // Accept-Encoding: gzip, deflate
  // Referer: http://10.0.0.79/
  // Connection: keep-alive
  // Upgrade-Insecure-Requests: 1
  // ContetTe plainxw-omrecded
  // Content-Length: 532  
  uint16_t nLen = 0, nPos = 0;
  String strLen,
          strContentLength = F("Content-Length: "); 

  nPos = strHTTPReq.indexOf(strContentLength);
  if (nPos >= 0)
  {
    strLen = strHTTPReq.substring(nPos + strContentLength.length(), strHTTPReq.length());
    nLen = strLen.toInt();
  }
  return nLen;
}

void CWifi::processHTTPRequest(WiFiEspClient& WifiClient)
{  
  boolean bNewLine = false;
  char cCh = 0;
  String strHTTPReq;

  debug("New HTTP request");
  m_nBytesRead = 0;

  while (WifiClient.connected()) 
  {
    if (WifiClient.available()) 
    {
      cCh = WifiClient.read();
      //Serial.print(cCh);

      strHTTPReq += String(cCh);

      // If you've gotten to the end of the line (received a newline
      // character) and the line is blank, the http request has ended,
      // so you can send a reply
      // A http request ends with a blank line
      if ((cCh == '\n') && bNewLine) 
      {
        debug(F("***********************************************************************"));
        debug(F("HTTP request:"));
        debug(strHTTPReq);
        debug(F("***********************************************************************"));
        
        // Default HTTP request
        // ---------------------
        // GET / HTTP/1.1
        // Host: 10.0.0.79
        // User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
        // Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*//*;q=0.8
        // Accept-Language: en-US,en;q=0.5
        // Accept-Encoding: gzip, deflate
        // Connection: keep-alive
        // Upgrade-Insecure-Requests: 1
        // Cache-Control: max-age=0
        
        if ((strHTTPReq.indexOf("GET / HTTP/1.1") >= 0) && !sendContent(WifiClient, false))
        {
          debug("Could not open default1.htm and/or default2.htm!");
        }
        else if ((strHTTPReq.indexOf("GET /favicon.ico HTTP/1.1") >= 0) && !sendFile(WifiClient, m_strFaviconFileName.c_str()))
        {
          debug("Could not open " + m_strFaviconFileName + "!");
        }
        else if ((strHTTPReq.indexOf("POST /program.htm HTTP/1.1") >= 0))
        {
          uint16_t nDataLen = getPostDataLength(strHTTPReq);

          if (!readPostData(WifiClient, nDataLen))
            debug("Could not save irrigation program to SD card!");
          WifiClient.flush();
          if (!sendContent(WifiClient, false))
            debug("Could not send submit form response!");
        }
        else if ((strHTTPReq.indexOf("POST /alarms.htm HTTP/1.1") >= 0))
        {
        }
        break;
      }
      // You're starting a new line
      if (cCh == '\n') 
        bNewLine = true;
      // You've gotten a character on the current line
      else if (cCh != '\r') 
        bNewLine = false;
    }
  }
  // give the web browser time to receive the data
  delay(100);

  // Close the connection:
  WifiClient.stop();
  debug("Client disconnected");
}

TextFile.cpp (1.86 KB)

Wifi.cpp (14.2 KB)

POST.TXT (83.8 KB)

TextFile.h (1.27 KB)

Wifi.h (1.33 KB)

I have more info for this problem.

I tried removing the SD card from the arduino, opening it in windows 7 and manually erased it.

I put it back in the arduino and did another form post in Firefox, then another one with different for settings.

I removed the SD card and opned it in Windows 7 again.

Attached is the file with my form data.

In my javascript for the HTML form I added a '~' character to denote the end of my data and I can see that character twice in the file data below.

So there appears to be two problems.

The first problem appears to be that creating a file with the arduino SD library with the O_WRITE | O_CREAT attibutes does not erase it - irritating! This goes against how file manipulation is implemented on other platforms.

So what is that you have to do in the SD library to erase the damn contents of a file?

The second problem appears to be that, when there is a lot of form data, firefox is sending it in multiple bursts and not in the fashion of sending continuous file data.
I.E. Because my form input tags 'Station1' to 'Station8' are repeated in each burst.

Also irritating.

How do I get around this second problem? How can I force Firefox to send the form data in a fashion similar to uploading a file?

Anyone have some suggestions?

POST.TXT (22.2 KB)