Go Down

Topic: opening txt file from spiffs in a browser (Read 273 times) previous topic - next topic

damo1023

Hello,

I have a ESP sensor which does log all system events to spiffs file (event log). I want to be able to open this file with a browser (to check for eventual system errors).

I have troubles sending this file to browser. It sends about 4-5 parts (each 1000bytes), and then it hangs. I guess there are too many parts sent in one session.

How would I send a long txt file to html page?


Thank you for your help!



Code is below:

Code: [Select]
void handleView_log()
{
  int siz;
  String oneline;
 
  wdt_reset();
  Serial.println (F("handleView_log"));
   
  server.sendHeader("Cache-Control", "no-cache, no-store, must-revalidate");
  server.sendHeader("Pragma", "no-cache");
  server.sendHeader("Expires", "-1");
  server.setContentLength(CONTENT_LENGTH_UNKNOWN); // *** BEGIN ***
  server.send(200, "text/html", "");
 
  s="";
  s=s+F("<!doctype html><html><head><title></title></head><body><h1>Event log</h1><form target=\"_blank\">");
  s=s+F("<table border=\"0\" cellpadding=\"1\" cellspacing=\"1\" style=\"width: 1200px;\"><tbody><p>");
  server.sendContent(s);
   
  log_file.close();
 
  log_file = SPIFFS.open("/log.txt", "r");
 
  if (!log_file)
  {
      s=s + F("<h1>Log file not found.</h1>");              // log.txt does not exist
  }
  else
  {
      if ( !log_file.size() )
      {
          s=s + F("<h1>Log file empty.</h1>");              // log.txt is empty
      }
      else
      {
          Serial.print F("Dumping log file size: ");        // log.txt is valid for sending to browser
          siz = log_file.size();
          Serial.println (siz);
          while (siz>0)                                     // siz will count unread/unsent bytes
          {
              s = "";
              while ( (s.length()<1000) & (siz>0) )           // read parts up to 1000bytes each
              {
                  oneline = log_file.readStringUntil('\n');
                  s = s + "<p>" + oneline + "</p>";           // add html handles for each line
                  siz = siz - oneline.length();
              }
              server.sendContent (s);                       // send part
          }
           
          s = F("</p><p></p><p></p>END OF LOG");
      }
  }
   
  s = s + F("</tbody></table>");
  server.sendContent(s);
  //Serial.println(s);
 
  server.sendContent(""); // *** END 1/2 ***
  server.client().stop(); // *** END 2/2 ***

  return;
}

vaj4088

You have posted a code snippet.  Code snippets can be a problem because they usually omit important information.

I would guess that the variable s is declared as String.  String is problematic and to be avoided on small memory devices such as most Arduinos.  String can lead to fragmented memory and hangs. C strings (text terminated by a null character) works well and avoids most hangs.

I am surprised that this even compiles:

Code: [Select]

          Serial.print F("Dumping log file size: ");        // log.txt is valid for sending to browser


I am away from the Arduino IDE and cannot try this on my mobile device.


damo1023

#2
Sep 25, 2018, 11:09 am Last Edit: Sep 25, 2018, 11:09 am by damo1023
Sorry to omit this info,

s is declared as global string: String s;

in start function I use s.reserve(2500); to make a room for it.


I am using ESP8266-12 in arduino IDE.

damo1023

I modified the code in a way it just reads chunks of file and prints millis and bytes remained to read:

Code: [Select]
void handleView_log()
{
  int siz;
  String oneline;
  
  wdt_reset();
  Serial.println (F("handleView_log"));
  log_file.close();
 
  log_file = SPIFFS.open("/log.txt", "r");
  
  if (!log_file)
  {
      s=s + F("<h2>Log file not found.</h2>");              // log.txt does not exist
  }
  else
  {
      if ( !log_file.size() )
      {
          s=s + F("<h2>Log file empty.</h2>");              // log.txt is empty
      }
      else
      {
          Serial.print F("Dumping log file size: ");        // log.txt is valid for sending to browser
          siz = log_file.size();
          Serial.println (siz);
          while (siz>0)                                     // siz will count unread/unsent bytes
          {
              s = "";
              while ( (s.length()<1000) & (siz>0) )           // read parts up to 2000bytes each
              {
                  oneline = log_file.readStringUntil('\n');
                  s = s + "<p>" + oneline + "</p>";           // add html handles for each line
                  siz = siz - oneline.length();
              }
              Serial.print (millis() );
              Serial.print ("      bytes left: ");
              Serial.println (siz);
          }
          
          s = F("</p><p></p><p></p><h3>END OF LOG</h3>");
      }
  }
  
  Serial.println(s);

  return;
}



It is clearly that I have a problem with spiffs read. It works ok for few ten times (takes some 100ms per chunk), than is slows down and takes app. 15 seconds per chunk.

Below is the output, first number is millis(), second number is bytes remained:

Code: [Select]
Dumping log file size: 37145
208022      bytes left: 36256
208095      bytes left: 35333
208163      bytes left: 34465
208234      bytes left: 33556
208300      bytes left: 32684
208369      bytes left: 31791
208436      bytes left: 30923
208508      bytes left: 30014
208574      bytes left: 29142
208646      bytes left: 28219
208715      bytes left: 27326
208787      bytes left: 26390
208860      bytes left: 25373
208934      bytes left: 24437
209002      bytes left: 23561
209070      bytes left: 22684
209143      bytes left: 21841
209214      bytes left: 20979
209286      bytes left: 20144
209360      bytes left: 19273
209438      bytes left: 18362
209517      bytes left: 17548
209591      bytes left: 16736
209668      bytes left: 15924
209746      bytes left: 15136
209823      bytes left: 14321
209899      bytes left: 13524
209973      bytes left: 12727
210049      bytes left: 11915
217327      bytes left: 11070
233087      bytes left: 10180
247612      bytes left: 9358
262224      bytes left: 8532
277243      bytes left: 7686
292277      bytes left: 6834
307250      bytes left: 5988
322273      bytes left: 5142
337307      bytes left: 4290
352032      bytes left: 3459
366945      bytes left: 2616
381918      bytes left: 1766
408856      bytes left: 1034



PaulS

Code: [Select]
          s = F("</p><p></p><p></p><h3>END OF LOG</h3>");

Stupid. It makes NO sense to take a string literal that is stored in flash memory and copy it into a String instance in SRAM.

Quit pissing away memory on Strings. The read() method is overloaded to read n characters into a char array. The array can then be NULL-terminated and used wherever the data is (somewhere outside of your useless snippet) actually sent to the client.
The art of getting good answers lies in asking good questions.

damo1023

#5
Sep 25, 2018, 02:44 pm Last Edit: Sep 25, 2018, 07:47 pm by damo1023
Sorry for my poor programming knowledge...

I am using string 's' just for some kind of buffer, to collect info and then send in a packet, not sending each few bytes separate.

Also I when I am sending large HTML file, I use this string as a buffer, it consists of HTML code and variables, measuring results etc. I willingly sacrificed 2500 bytes of SRAM to have this buffer. I know it is not such a good idea, but still maybe is better this way, than using hundreds of calls to "server.sendContents (xxx).


I think I found the reason for above slow reading: I have to upgrade spiffs library to newer. I upgraded spiffs to 0.3.7.

After some code tweaking here is code that works:

Code: [Select]
void handleView_log()
{
  int siz;
  String oneline;
  char kar;
 
  wdt_reset();
  Serial.println (F("handleView_log"));
   
  server.sendHeader("Cache-Control", "no-cache, no-store, must-revalidate");
  server.sendHeader("Pragma", "no-cache");
  server.sendHeader("Expires", "-1");
  server.setContentLength(CONTENT_LENGTH_UNKNOWN); // *** BEGIN ***
  server.send(200, "text/html", "");
 
  s = F("<!doctype html><html><h1>Event log</h1>");
  server.sendContent(s);
   
  log_file.close();
  delay(100);
 
  log_file = SPIFFS.open("/log.txt", "r");
 
  if (!log_file)
  {
      server.sendContent F("<h2>Log file not found.</h2>");              // log.txt does not exist
  }
  else
  {
      if ( !log_file.size() )
      {
          server.sendContent F("<h2>Log file empty.</h2>");              // log.txt is empty
      }
      else
      {
          Serial.print F("Dumping log file size: ");        // log.txt is valid for sending to browser
          siz = log_file.size();
          Serial.println (siz);
          while (siz>0)                                     // siz will count unread/unsent bytes
          {
              wdt_reset();
              s = "";
              while ( s.length()<2000 )                     // read parts up to 2000bytes each
              {
                  if (!log_file.available())
                  {
                      siz=0;
                  }
                  else
                  {
                      kar = log_file.read();
                      switch (kar)
                      {
                          case (10): s = s + "</p>";
                                       break;
                          case (13): s = s + "<p>";
                                       break;
                          default: s = s + kar;
                      }
                      siz--;
                  }
                 
                 
                  if (siz<=0) break;
              }
              Serial.print (millis() );
              Serial.print ("      bytes left: ");
              Serial.println (siz);
              server.sendContent (s);                       // send part
          }
           
      }
  }
   
  server.sendContent F("</p><p></p><p></p><h3>END OF LOG</h3>");
  wdt_reset();
  server.sendContent (""); // *** END 1/2 ***
  server.client().stop(); // *** END 2/2 ***
 
  return;
}

PaulS

Quote
I am using string 's' just for some kind of buffer,
No. You are pissing away resources on Strings. They are NOT the same as strings, in any way, shape, or form.

Quote
to collect info and then send in a packet, not sending each few bytes separate.
You don't think the client is smart enough to do that?

Quote
I know it is not such a good idea, but still maybe is better this way, than using hundreds of calls to "server.sendContents (xxx).
Why would you need hundreds of them? One, called in a loop, sending the contents of the string (NOT String) of data read from the file is all that is needed.
The art of getting good answers lies in asking good questions.

damo1023

#7
Sep 26, 2018, 07:18 am Last Edit: Sep 26, 2018, 08:09 am by damo1023
Hello,

I would gladly leave Strings, but have no idea how.

Here is a part of my code that serves html page. I am combining fixed text (HTML code) and measured values ( d[ x ].value ) and variables ( d[ x ].tag etc.). I don't know how else to do that, that's why I using String.

I use String s as a global, reserving a space of 2500. I assume by that, I sacrifized 2500 bytes of RAM for that kind of jobs in my project. Am I wrong?

Is there a way to see, what is "eating" my RAM, like some kind of RAM table?

I would appreciate if you can point me how to achieve above task without using Strings.

Here is the part of code (I did not attached whole HTML since it is very long:

Code: [Select]

 server.sendHeader("Cache-Control", "no-cache, no-store, must-revalidate");
 server.sendHeader("Pragma", "no-cache");
 server.sendHeader("Expires", "-1");
 server.setContentLength(CONTENT_LENGTH_UNKNOWN); // *** BEGIN ***
 server.send(200, "text/html", "");
 
 s="";
 s=s+ F("<h4>Current measured values:</h4>");
 s=s+F("<table><tbody>");
 s=s+F("<tr><td>") + String(d[0].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[0].value) + ("</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>");
 s=s+F("<td>") + String(d[4].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[4].value)+F("</td></tr>");

 s=s+F("<tr><td>") + String(d[1].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[1].value) + ("</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>");
 s=s+F("<td>") + String(d[5].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[5].value)+F("</td></tr>");

 s=s+F("<tr><td>") + String(d[2].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[2].value) + ("</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>");
 s=s+F("<td>") + String(d[6].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[6].value)+F("</td></tr>");

 s=s+F("<tr><td>") + String(d[3].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[3].value) + ("</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>");
 s=s+F("<td>") + String(d[7].tag) + F("</td><td>&nbsp;&nbsp;&nbsp;</td><td>") + String(d[7].value)+F("</td></tr>");
 s=s+F("</table></tbody><p>&nbsp;</p>");

 s=s+ F("<h4>Enter setup page:</h4>");
 s=s+F("<table><tbody><tr><td>User</td> <td><input type=\"text\" name=\"username\"></td></tr>");
 s=s+F("<tr><td>Password:</td> <td><input type=\"password\" name=\"password\"></td></tr></table>");
 s=s+F("<p>&nbsp;</p><input type=\"submit\" value=\"Login\"></form> ");

 server.sendContent(s);

..... some more HTML in String and sending with server.sendContent(s);

..... some more HTML in String and sending with server.sendContent(s);

....




 server.sendContent(""); // *** END 1/2 ***
 server.client().stop(); // *** END 2/2 ***




vaj4088

I suggest that you go to www.cplusplus.com and use the search box to look for the strcat(...) function.  This function and others like it are available on the Arduino and make it easy to use c strings instead of String.

It would be easy for me to convert your code to use strcat(...) but I am hoping that you are willing to do some digging and get educated.  There may be a little pain at first but in the long run it will be worthwhile.


damo1023

#9
Sep 26, 2018, 08:37 am Last Edit: Sep 26, 2018, 08:47 am by damo1023
Thank you for reply.

Of course I am not expecting you to do a job for me, I just need some guidance and want to learn to do it.  :)


If I understand correctly, the right way is to have a c string (char array) and do the same thing I did with String?

i.e.:
Code: [Select]

unsigned char s[1000];

strcat (s, "<h4>Current measured values:</h4>");
strcat (s, "<table><tbody>");
strcat (s, "<tr><td>");
strcat (s, d[0].tag);
strcat (s, "</td><td>&nbsp;&nbsp;&nbsp;</td><td>");
strcat (s, d[0].value);
...
...
...
...




vaj4088

No.  Almost but not quite.

At the start, the variable s is not a c string.  The example for strcat(...) at www.cplusplus.com starts by using strcpy(...) which is one way to get a c string started.  Another way would be

s[0] = 0 ;

Also, the parentheses around d[0].tag and d[0].value are harmless but unnecessary.

vaj4088

By the way, I think that if the variable s is declared globally, s may already be a zero-length c string, but not if s is declared in a function.

From your example, I could not tell where s was declared and I took the safer route.

UKHeliBob

No.  Almost but not quite.

At the start, the variable s is not a c string.  The example for strcat(...) at www.cplusplus.com starts by using strcpy(...) which is one way to get a c string started.  Another way would be

s[0] = 0 ;
You don't need to "get a c string started", just declare it large enough to hold the longest string that will be put in it.
Code: [Select]

char text[30];

void setup()
{
  Serial.begin(115200);
  strcat(text, "Hello World");
  Serial.println(text);
  strcat(text, " - anyone there ?");
  Serial.println(text);
}

void loop()
{
}
Please do not send me PMs asking for help.  Post in the forum then everyone will benefit from seeing the questions and answers.

vaj4088

That is fine if the variable is global but not if it is stack-based.


UKHeliBob

Declaring it static will get round that problem
Please do not send me PMs asking for help.  Post in the forum then everyone will benefit from seeing the questions and answers.

Go Up