NodeMCU 8266 write to spiffs: incorrect characters

Using Arduino Programming Questions
1

I have a web connection monitor that pings a site with two pings every 60 sec and records the ping metrics to spiffs that is (infrequently) accessed as required via FTP. Example data from SPIFFS is shown below.
For some reason it SOMETIMES inserts substitute characters at the end of a line. So instead of
; LF (ASCII 3b 20 0a) it records ÿÿÿ (ff ff ff)
It seems to be consistent in that IF its wrong its always the last three characters of the line that are incorrect so (below) *ÿÿÿ SHOULD have been ***LF
ADDED: When I send the fStatsLine to the serial monitor it prints correctly - but the line in SPIFFS is still incorrect.

any ideas?

Date, Time; WiFIStatus%; lost, wait;
2022/03/10, 11:43:34; 100; 0.00, 0.33;
2022/03/10, 11:46:34; ; , ; *** netLogger restart - mains failure? *ÿÿÿ2022/03/10, 11:46:34; ; , ; *** web available ***
2022/03/10, 11:47:46; ; , ; *** netLogger restart - mains failure? ***
2022/03/10, 11:47:46; ; , ; *** web available ***
2022/03/10, 12:46:49; 100; 0.00, 21.45, 2.31ÿÿÿ2022/03/10, 13:46:50; 100; 0.00, 21.17, 2.09;
2022/03/10, 14:46:50; 100; 0.00, 21.53, 2.06;
2022/03/10, 15:46:50; 100; 0.00, 21.37, 1.54;
2022/03/10, 16:46:50; 100; 0.00, 21.63, 2.54ÿÿÿPinging 185.217.104.171 test server
Date, Time; WiFIStatus%; lost, wait, sigma;
2022/03/10, 17:11:21; 100; 0.00, 0.33, 2.56;
2022/03/10, 18:11:22; 100; 0.00, 21.72, 1.92;
2022/03/10, 19:11:23; 100; 0.00, 22.75, 3.39;
2022/03/10, 20:11:22; 100; 0.02, 25.05, 11.66;
2022/03/10, 21:11:22; 100; 0.00, 23.17, 3.03ÿÿÿ2022/03/10, 22:11:23; 100; 0.00, 23.45, 2.86;
2022/03/10, 23:11:23; 100; 0.00, 23.85, 3.34ÿÿÿPinging 185.217.104.171 test server
Date, Time; WiFIStatus%; lost, wait, sigma;
2022/03/11, 00:00:22; 100; 0.00, 20.88, 10.81;
2022/03/11, 01:00:23; 100; 0.00, 25.70, 6.18;
2022/03/11, 02:00:23; 100; 0.00, 27.18, 5.21;
2022/03/11, 03:00:23; 100; 0.00, 27.02, 4.80;
2022/03/11, 04:00:23; 100; 0.00, 28.45, 4.99ÿÿÿ2022/03/11, 05:00:23; 100; 0.02, 27.68, 12.36;
2022/03/11, 06:00:23; 100; 0.00, 28.30, 15.17;

The string to be printed is prepared in "prepStats()" and printed with "writeFile()"

void prepStats() {
  //prepare header lines for filing Dont use colons because then date gets separated
  //prepare stats line for filing - about 50 chars.
  //fHeadLine:  Pinging 107.162.133.62 test server
  //fHeadLine1: Date, Time; WiFIStatus%; lost, wait;
  //fStatsLine: 05/09/2021, 09:00:03; 100; 0.00, 1.90;

  if(diagnostic) Serial.println("prepStats: Stats for filing on SPIFFS:  ");
  fHeadLine = "Pinging ";
  fHeadLine += host1[6].ipString;
  fHeadLine += testServer;
  fHeadLine += "\n";
  fHeadLine1 = "Date, Time; WiFIStatus%; lost, wait, sigma;";
  fHeadLine1 += "\n";
  //prepare total loss values and ping averages - force excel to see as floats
  host1[6].nAvPing = (pingLossTotals[6] / nRounds);
  host1[6].tAvPing = (tPingTotals[6] / nRounds);
  fStatsLine = "";
  fStatsLine += myFormattedDate;
  fStatsLine += "; ";
  //here WiFiUp is just a number; convert to percentage & add to fStatsLine
  WiFiUp = WiFiUp * 100;
  WiFiPercent = WiFiUp / nRounds;
  fStatsLine += WiFiPercent;
  fStatsLine += "; ";
  //add lost & wait to fStatsLine
  fStatsLine += host1[6].nAvPing;  //total lost pings
  fStatsLine += ", ";
  fStatsLine += host1[6].tAvPing;  //average ping time
  fStatsLine += ", ";
  //*** calculate sigma and add to stats line
  float temp = host1[6].tSp;
  temp *= temp;
  temp = temp / nRounds;
  temp = host1[6].tSpSq - temp;
  temp = temp / nRounds;
  sigma = sqrt(temp);
  fStatsLine += sigma;
  fStatsLine += "; \n";  // END OF STATS LINE WITH ASCII 3b 20 0a
}

void writeFile() {  //write stats; and header line only if its a new day or first readings
  int bytesWritten;
  String fileName = "/" + sYear + "-" + sMonth + ".txt"; //filename MUST be fully qualified ie have a leading slash.
  File file = SPIFFS.open(fileName, "a"); //"w" to write a file, "a" to append
  if (!file) {
    Serial.println("Error opening file for writing");
    return;
  }
  if (DD != today) { //write headline only if its a new day
    bytesWritten = file.print(fHeadLine);
    bytesWritten = file.print(fHeadLine1);
    today = DD;
  }
  //write stats
  bytesWritten = file.print(fStatsLine);
  file.close();
}
2

Did you try running Serial slower?

A little tip: there is no need to build a single string or String to print or write to file.
Just print or write the pieces since they all assemble in an output buffer anyway,
so why do the extra step that uses RAM and CPU cycles for no gain whatsoever?

3

Thanks @GoForSmoke I'll try that tip. When the program is running the diagnostics are off and no serial printing occurs;
As you will have seen the print to file only occurs at hourly intervals so speed isnt an issue; however the serial is still running so I'll try dropping the baud rate anyway.

4

I've seen those kind of errors in serial run too fast. The UART misses bits.

However you read the "ping", that's where the bad data is collected.

Something else... Arduino serial output buffer is 64 chars. If your code tries to overfill it, serial chars have to be sent before any other code runs.

I'm guessing that the speed of your code getting the ping data is what matters most.

5

No, I've dropped the serial baud rate from 115200 to 19200 and if anything its worse. I've also added a 1ms delay here - no improvement.

//write stats
  bytesWritten = file.print(fStatsLine);
delay(1);l
  file.close();
6

That supports buffer overfills. The buffer empties slower at 19200.

What are you reading in the ping? How read the ping?

Are your files on SD?

7 
#include <Pinger.h>                 //handles the ping fuctions
#include <FS.h>                     //provides the SPIFFS library
#include <ESP8266FtpServer.h>       //ftp server library

So I'm getting the ping data from the pinger.h library.
Files are stored in the ESP8266 flash memory.

I hadnt realised the SPIFFS used the external SPI interface but there are no connections to those pins in my circuit.

Interestingly: when I increased the rate at which data is written to spiffs
(from 1 write per hour to 1 write per minute) I observe much more frequent errors.

2022/03/12, 05:22:07; 100; 0.00, 30.00, 3.82ÿÿÿ2022/03/12, 05:23:07; 100; 0.00, 29.10, 6.02;
2022/03/12, 05:24:07; 100; 0.00, 27.30, 4.67;
2022/03/12, 05:25:07; 100; 0.00, 28.50, 4.88ÿÿÿ2022/03/12, 05:26:07; 100; 0.00, 28.30, 4.12ÿÿÿ2022/03/12, 05:27:07; 100; 0.00, 29.30, 5.60;
2022/03/12, 05:28:07; 100; 0.00, 28.20, 4.71;
2022/03/12, 05:29:07; 100; 0.00, 28.50, 6.00;
2022/03/12, 05:30:07; 100; 0.00, 29.30, 3.72ÿÿÿ2022/03/12, 05:31:07; 100; 0.00, 28.00, 4.22ÿÿÿ2022/03/12, 05:32:07; 100; 0.00, 29.00, 4.22;
2022/03/12, 05:33:07; 100; 0.00, 28.40, 6.04;
2022/03/12, 05:34:07; 100; 0.00, 26.40, 5.16;
2022/03/12, 05:35:07; 100; 0.00, 25.20, 4.07ÿÿÿ2022/03/12, 05:36:07; 100; 0.00, 27.60, 5.43ÿÿÿ2022/03/12, 05:37:07; 100; 0.00, 30.00, 4.43;
2022/03/12, 05:38:07; 100; 0.00, 29.70, 3.85;
2022/03/12, 05:39:07; 100; 0.00, 28.10, 4.66;
2022/03/12, 05:40:07; 100; 0.00, 32.90, 7.91ÿÿÿ2022/03/12, 05:41:07; 100; 0.00, 25.40, 4.39ÿÿÿ2022/03/12, 05:42:07; 100; 0.00, 29.60, 3.23;

I wondered if (accessing the data via FTP / conflicting with writes) was causing the problem but during the sample shown here there was no FTP access.

8

I didn't know that 8266 uses flash as a drive. Don't treat it like magnetic storage!
For example never sort a file, write a sorted index file into the data, as many as needed.

How about printing less to file? Start every line with millis() in HEX, a nice uniform 8 chars.
Date and time are in the header which begins with a hex time stamp, a synch spot.
Interval between header and data lines is end time minus start time in ms.

9

Going through the code and output, the error is 3 chars that should be 3b 20 0a.

  fStatsLine += "; \n";  // END OF STATS LINE WITH ASCII 3b 20 0a
10

Well, fingers firmly crossed I seen to have fixed it.
Lines that did not show errors had eg
fHeadLine += "\n";
lines that did had eg
fStatsLine += "; \n";

I've edited the code so every built string ends in spaces, and adds the EOL (0a) using +="\n";

and so far no errors - its too early to say its fixed, but definitely better.

I dont understand why SPIFFS has problems with a string ending "; \n".

I've also added a delay before closing the spiffs file.

//write stats
  bytesWritten = file.print(fStatsLine);
  delay(10);
  file.close();

void prepStats() {
  //prepare header lines for filing Dont use colons because then date gets separated
  //prepare stats line for filing - about 50 chars.
  //fHeadLine:  Pinging 107.162.133.62 test server
  //fHeadLine1: Date, Time; WiFIStatus%; lost, wait;
  //fStatsLine: 05/09/2021, 09:00:03; 100; 0.00, 1.90;

  //if(diagnostic) Serial.println("prepStats: Stats for filing on SPIFFS:  ");
  fHeadLine = "Pinging ";
  fHeadLine += host1[6].ipString;
  fHeadLine += testServer;
  fHeadLine += "\n";
  fHeadLine1 = "Date, Time; WiFIStatus%; lost, wait, sigma;";
  fHeadLine1 += "\n";
  //prepare total loss values and ping averages - force excel to see as floats
  host1[6].nAvPing = (pingLossTotals[6] / nRounds);
  host1[6].tAvPing = (tPingTotals[6] / nRounds);
  fStatsLine = "";
  fStatsLine += myFormattedDate;
  fStatsLine += "; ";
  //here WiFiUp is just a number; convert to percentage & add to fStatsLine
  WiFiUp = WiFiUp * 100;
  WiFiPercent = WiFiUp / nRounds;
  fStatsLine += WiFiPercent;
  fStatsLine += "; ";
  //add lost & wait to fStatsLine
  fStatsLine += host1[6].nAvPing;  //total lost pings
  fStatsLine += ", ";
  fStatsLine += host1[6].tAvPing;  //average ping time
  fStatsLine += ", ";
  //*** calculate sigma and add to stats line
  float temp = host1[6].tSp;
  temp *= temp;
  temp = temp / nRounds;
  temp = host1[6].tSpSq - temp;
  temp = temp / nRounds;
  sigma = sqrt(temp);
  fStatsLine += sigma;
  fStatsLine += "   ";
  fStatsLine += "\n";
  if(diagnostic){
  //Serial.print(fHeadLine);
  //Serial.print(fHeadLine1);
  Serial.print(fStatsLine);
  }
}

void writeFile() {  //write stats; and header line only if its a new day or first readings
  int bytesWritten;
  String fileName = "/" + sYear + "-" + sMonth + ".txt"; //filename MUST be fully qualified ie have a leading slash.
  File file = SPIFFS.open(fileName, "a"); //"w" to write a file, "a" to append
  if (!file) {
    Serial.println("Error opening file for writing");
    return;
  }
  if (DD != today) { //write headline only if its a new day
    bytesWritten = file.print(fHeadLine);
    bytesWritten = file.print(fHeadLine1);
    today = DD;
  }
  //write stats
  bytesWritten = file.print(fStatsLine);
  delay(10);
  file.close();
}

void logRestart() {  //write restart message to file
  int bytes;
  String fileName = "/" + sYear + "-" + sMonth + ".txt"; //filename MUST be fully qualified ie have a leading slash.
  File file = SPIFFS.open(fileName, "a"); //"w" to write a file, "a" to append
  if (!file) {
    Serial.println("Error opening file for writing");
    return;
  }
  String restart = ""; 
  restart += myFormattedDate;  //match format for fStatsLine
  restart += ";    ;     ,      ; *** netLogger restart - mains failure? ***  ";
  restart += "\n";
  bytes = file.print(restart);
  Serial.print(restart);
  delay(10);
  file.close();
}

void logWebState() {  //write web failure message to file
  int bytes;
  String fileName = "/" + sYear + "-" + sMonth + ".txt"; //filename MUST be fully qualified ie have a leading slash.
  File file = SPIFFS.open(fileName, "a"); //"w" to write a file, "a" to append
  if (!file) {
    Serial.println("Error opening file for writing");
    return;
  }
  String webState = "";
  webState += myFormattedDate;  //match format for fStatsLine
  if (webGood == 0) webState += ";    ;     ,      ; *** web down ***  ";   else webState += ";    ;     ,      ; *** web available ***  ";
  webState += "\n";
  bytes = file.print(webState);
  Serial.print(webState);
  file.close();
}
11

Agreed. Fixed but not explained.

Did you try file.flush(); before file.close(); ?

(Actually I suspect file.close(); executes file.flush();.)

12

Well, it seems stable now. An interesting observation - until this was fixed access to spiffs via FTP was problematic - the file system was displayed immedately
image
but there was often a long delay before it would download a file.

Since the problem has been resolved the download is almost immediate.

13

I've got the same problem, though sometimes just ÿÿ instead of ÿÿÿ.
I'd already tried the delay() before files.close() to no avail so it's probably not that by itself (it goes into deep sleep after closing so I wondered if there was some asynchronous activity that didn't finish). I've kept in a small delay anyway.
The SPI talk did prompt me to try something which so far is working (been here before though !).
I'm using an ESP12F in a burner board which connects to the TX and RX plus GPI00, GPI02 as I expect any usb interface does. Running without the board (but now I can't watch the serial monitor) it's behaving properly. If it comes back I will update this post.
Update: It's still there even with a space at the end of each record.
Update 2: I switched to LittleFS and there are no unwanted characters now but I cannot find an FTP library that will work alongside softAP. ESPFtpServer does work with LittleFS and a non SoftAP WiFi connection.

14

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.