[SOLVED] setTimeout(), parseInt() and file reading from sd card

Projects Programming
1

Hi.

I am using parseInt() to get some values froms a txt file.

Every thing is working but I still have that lag (delay) from the parseInt() function.

I have found that many are using setTimeout() to reduce that lag, but all of them are using Serial.setTimeout(), and I can't find any example showing how to use it for files stored on sd card.

Is there a way to use it in such case?

My files have always a fixed numbre of columns ex:

//for this example i would use 4 parsInt() to get the 4 values of every line at the same time
//i found that it to be simple and easy to work with that way

1;50;205;25;
663;744;88;52;
0;25;11;1;
20;550;50;22;

Or is there a better (without the delay) way to get data from such files?

2

Is there a way to use it in such case?

parseInt() is a member of a class. Which class instance are you using to invoke parseInt()?

Reading an integer from a stream stops when the first non-digit (except a - sign) is found. So, you can make reading an integer almost instantaneous by making sure that there is something (like a space) following the integer in the stream.

As always, POST YOUR CODE.

3

Hi PaulS :smiley:

This is the function I am using to plot some graphs:

//******************************************************************************************************************************
void drawcurve (int  prm, String filex) {
  tft.fillScreen(TFT_BLACK);
  affgraph = 1;
  float maxv = -32767;//was using integers
  float minv = 32767;
  myFile = SD.open(filex);
  if (myFile) {
    int zz = 0;
    while (myFile.available()) {
      int w1 = myFile.parseInt();
      int x1 = myFile.parseInt();
      int y1 = myFile.parseInt();
      int z1 = myFile.parseInt();
      switch (prm) {
        case 1:
          zz = x1; break;
        case 2:
          zz = y1; break;
        case 3:
          zz = z1; break;
      }
      if (w1 != 0 || x1 != 0 || z1 != 0 || y1 != 0) {
        if ((zz / 100) > maxv) {
          maxv = (zz / 100);
        }
        if ((zz / 100) < minv) {
          minv = (zz / 100);
        }
      }
    }
  }
  myFile.close();
  myFile = SD.open(filex);
  if (myFile) {
    float yy = 0;
    while (myFile.available()) {
      int w2 = myFile.parseInt();
      int x2 = myFile.parseInt();
      int y2 = myFile.parseInt();
      int z2 = myFile.parseInt();
      switch (prm) {
        case 1:
          yy = x2; break;
        case 2:
          yy = y2; break;
        case 3:
          yy = z2; break;
      }
      int minvm = minv;
      int maxvm = maxv;
      maxvm = maxvm + 1;
      float ech = (maxv + 1 - minv) / 5;
      //  yy=x;
      if (w2 != 0 || x2 != 0 || z2 != 0 || y2 != 0) {
        Graph( w2, yy / 100, 40, 260, 400, 200, 0, 60, 10, minvm, maxvm, ech, "TC", "t", "temp", TFT_BLUE, TFT_WHITE, TFT_WHITE, TFT_WHITE, TFT_BLACK, affgraph);//custom plotting function
      }
    }
    myFile.close();
  }
  else {
    TFTPrint("can't find " + filex, 0, 150, 2, 2, TFT_BLACK, TFT_RED);
  }
}
//**********************************************************************************************************

I first choose the parameter i want. Then arduino looks into the file twice:

  • 1 to look for max and min values of the parameter I chose (for scaling)
  • 2 to plot the graphic

The code is working, but i have about 3s delay before I get the graph (reading file?), and another 3s after gettinng the graph (timeout?).

4

The code is working, but i have about 3s delay before I get the graph (reading file?), and another 3s after gettinng the graph (timeout?).

Add some Serial.print() statements in the code, and print the value of millis() after a suitable prefix, to show where the code is spending the time. I don't think that you'll find that it is in the parseInt() function.
Opening and closing files is not instantaneous. Calling the Graph() function probably isn't, either.

5

PaulS:
Add some Serial.print() statements in the code, and print the value of millis() after a suitable prefix, to show where the code is spending the time. I don't think that you'll find that it is in the parseInt() function.
Opening and closing files is not instantaneous. Calling the Graph() function probably isn't, either.

I will try that, thanks.
What i'm worried about is the delay after the graph is plotted (the file is closed and the Graph() function has finished its tastk).
I read that parseInt() has a defaut time out of 1000ms, and using setTimeout (well, it is Serial.setTimeout() as every example i saw was about serial input) can be used to lower that delay (ex: Serial.setTimeout(10);).

6

I read that parseInt() has a defaut time out of 1000ms, and using setTimeout (well, it is Serial.setTimeout() as every example i saw was about serial input) can be used to lower that delay (ex: Serial.setTimeout(10)

The setTimeout() method is actually part of the same class that parseInt(), etc. are in. So, myFile.setTimeout() will work.

But, before you set it to a small value, try setting it to a large value, and see if you see any change. I do not think that parseInt() is what is taking the time. If you set the time out value to 10 seconds (the default is one second), and the program takes no longer, then setting it to a smaller value will not speed the program up.

7

PaulS:
The setTimeout() method is actually part of the same class that parseInt(), etc. are in. So, myFile.setTimeout() will work.

But, before you set it to a small value, try setting it to a large value, and see if you see any change. I do not think that parseInt() is what is taking the time. If you set the time out value to 10 seconds (the default is one second), and the program takes no longer, then setting it to a smaller value will not speed the program up.

Hi again. I tried that, but nothing changed. So I tried the millis() method you suggested:

  • First, I called the Serial.println(millis()) just when the fuction starts, just before calling the file to open, just after that, just before calling the file to close and just after that. And the result is:
millis just when draw curve starts :
23486
millis before first file opening:
23529
millis after first file opening:
23555
millis just before first file close:
27621
millis before second file opening:
27637
millis after second file opening:
27680
millis before second file closing:
31795
millis after second file closing:
31808

After that i can say that calling the function, the file to open or to close does not take much time.

So I called the Serial.println(millis()) after eatch file treatment (inside while(myFile.available())) and I got this :

millis just when draw curve starts :
16321
millis before first file opening:
16364
millis after first file opening:
16389
millis after every line treatement (file1):
16433
millis after every line treatement (file1):
16487
millis after every line treatement (file1):
16541
millis after every line treatement (file1):
16595
millis after every line treatement (file1):
16649
millis after every line treatement (file1):
16703
millis after every line treatement (file1):
16757
millis after every line treatement (file1):
16812
millis after every line treatement (file1):
16866
millis after every line treatement (file1):
16919
millis after every line treatement (file1):
16973
millis after every line treatement (file1):
17028
millis after every line treatement (file1):
17082
millis after every line treatement (file1):
17136
millis after every line treatement (file1):
17189
millis after every line treatement (file1):
17244
millis after every line treatement (file1):
17298
millis after every line treatement (file1):
17352
millis after every line treatement (file1):
17406
millis after every line treatement (file1):
17461
millis after every line treatement (file1):
17514
millis after every line treatement (file1):
17568
millis after every line treatement (file1):
17623
millis after every line treatement (file1):
17677
millis after every line treatement (file1):
17731
millis after every line treatement (file1):
17784
millis after every line treatement (file1):
17839
millis after every line treatement (file1):
17893
millis after every line treatement (file1):
17947
millis after every line treatement (file1):
18001
millis after every line treatement (file1):
22009
millis just before first file close:
22035
millis before second file opening:
22080
millis after second file opening:
22123
millis after every line treating and plotting:
22180
millis after every line treating and plotting:
22238
millis after every line treating and plotting:
22295
millis after every line treating and plotting:
22352
millis after every line treating and plotting:
22410
millis after every line treating and plotting:
22466
millis after every line treating and plotting:
22523
millis after every line treating and plotting:
22581
millis after every line treating and plotting:
22638
millis after every line treating and plotting:
22695
millis after every line treating and plotting:
22753
millis after every line treating and plotting:
22810
millis after every line treating and plotting:
22866
millis after every line treating and plotting:
22924
millis after every line treating and plotting:
22981
millis after every line treating and plotting:
23038
millis after every line treating and plotting:
23096
millis after every line treating and plotting:
23153
millis after every line treating and plotting:
23211
millis after every line treating and plotting:
23267
millis after every line treating and plotting:
23324
millis after every line treating and plotting:
23382
millis after every line treating and plotting:
23439
millis after every line treating and plotting:
23496
millis after every line treating and plotting:
23554
millis after every line treating and plotting:
23611
millis after every line treating and plotting:
23667
millis after every line treating and plotting:
23725
millis after every line treating and plotting:
23782
millis after every line treating and plotting:
23839
millis after every line treating and plotting:
27846
millis before second file closing:
27873
millis after second file closing:
27917

As you can see, there is a delay (about 4000ms) between the treatment last line treatment and the file closing. (I am calling parseInt 4 times * 4 timeout = 4000ms?).

What do you think?.

(This is the code with millis() printing: )

void drawcurve (int  prm, String filex) {
  Serial.println("millis just when draw curve starts :");
  Serial.println(millis());
  tft.fillScreen(TFT_BLACK);
  //TFTPrint("thisparam:"+String(prm), 240, 160, 2, 2, TFT_WHITE, TFT_RED);
  affgraph = 1;
  float maxv = -32767;//was using integers
  float minv = 32767;
  Serial.println("millis before first file opening:");
  Serial.println(millis());
  myFile = SD.open(filex);
  Serial.println("millis after first file opening:");
  Serial.println(millis());
  if (myFile) {
    int zz = 0;
    while (myFile.available()) {
      int w1 = myFile.parseInt();
      int x1 = myFile.parseInt();
      int y1 = myFile.parseInt();
      int z1 = myFile.parseInt();
      switch (prm) {
        case 1:
          zz = x1; break;
        case 2:
          zz = y1; break;
        case 3:
          zz = z1; break;
      }
      if (w1 != 0 || x1 != 0 || z1 != 0 || y1 != 0) {
        if ((zz / 100) > maxv) {
          maxv = (zz / 100);
        }
        if ((zz / 100) < minv) {
          minv = (zz / 100);
        }
      }
      Serial.println("millis after every line treatement (file1):");
      Serial.println(millis());
    }
  }
  Serial.println("millis just before first file close:");
  Serial.println(millis());
  myFile.close();
  Serial.println("millis before second file opening:");
  Serial.println(millis());
  myFile = SD.open(filex);
  Serial.println("millis after second file opening:");
  Serial.println(millis());
  if (myFile) {
    float yy = 0;
    while (myFile.available()) {
      int w2 = myFile.parseInt();
      int x2 = myFile.parseInt();
      int y2 = myFile.parseInt();
      int z2 = myFile.parseInt();
      switch (prm) {
        case 1:
          yy = x2; break;
        case 2:
          yy = y2; break;
        case 3:
          yy = z2; break;
      }
      int minvm = minv;
      int maxvm = maxv;
      maxvm = maxvm + 1;
      float ech = (maxv + 1 - minv) / 5;
      //  yy=x;
      if (w2 != 0 || x2 != 0 || z2 != 0 || y2 != 0) {

        Graph( w2, yy / 100, 40, 260, 400, 200, 0, 60, 10, minvm, maxvm, ech, "Light", "Time", "L it", TFT_BLUE, TFT_WHITE, TFT_WHITE, TFT_WHITE, TFT_BLACK, affgraph);

        // TFTPrint(String(maxv), 100, 150, 2, 2, TFT_BLACK, TFT_RED);
        // TFTPrint(String(minv), 100, 180, 2, 2, TFT_BLACK, TFT_RED);
        //  TFTPrint(String(minvm), 100, 210, 2, 2, TFT_BLACK, TFT_RED);
        //   TFTPrint(String(maxvm), 100, 240, 2, 2, TFT_BLACK, TFT_RED);
        //     TFTPrint(String(ech), 100, 270, 2, 2, TFT_BLACK, TFT_RED);
      }
      Serial.println("millis after every line treating and plotting:");
      Serial.println(millis());
    }
    Serial.println("millis before second file closing:");
    Serial.println(millis());
    myFile.close();
    Serial.println("millis after second file closing:");
    Serial.println(millis());
  }
  else {
    TFTPrint("can't find " + filex, 0, 150, 2, 2, TFT_BLACK, TFT_RED);
  }
}
8

Hi again. Solved :smiley:

myFile.setTimeout() solved it, I just tried it at first in the loop function or after file oppening and it didn't work.

I tried it just before starting myFile.parseInt() and it worked :slight_smile:

Thank you for the help PaulS ^^

9

Apparently, the last record of your file does not contain a carriage return and/or line feed.

10

PaulS:
Apparently, the last record of your file does not contain a carriage return and/or line feed.

Hi PaulS. I don't know what is carriage return or line feed :smiley:

11

fjtheknight:
Hi PaulS. I don't know what is carriage return or line feed :smiley:

RogerPate_stackoverflow:
Carriage return means to return to the beginning of the current line without advancing downward. The name comes from a printer's carriage, as monitors were rare when the name was coined. This is commonly escaped as "\r", abbreviated CR, and has ASCII value 13 or 0x0D.

Linefeed means to advance downward to the next line; however, it has been repurposed and renamed. Used as "newline", it terminates lines (commonly confused with separating lines). This is commonly escaped as "\n", abbreviated LF or NL, and has ASCII value 10 or 0x0A. CRLF (but not CRNL) is used for the pair "\r\n".

Form feed means advance downward to the next "page". It was commonly used as page separators, but now is also used as section separators. (It's uncommonly used in source code to divide logically independent functions or groups of functions.) Text editors can use this character when you "insert a page break". This is commonly escaped as "\f", abbreviated FF, and has ASCII value 12 or 0x0C.

I know now what they are, but still don't know how to use them.

12

[When you typed the data into the file,
at the end of the first three
records, you hit the enter key, to add a carriage return and line feed.
On the last record, you appear not to have.]

So, when reading the data, there is nothing after the last digit of the last value to tell parseInt() to stop reading. So, it is sitting around, hoping that more data will appear in the file.

Edit the file. At the end of the last record, hit the enter key, so that there are 4 CRs and 4 LFs in the file, not just three of each.

13

PaulS:
[When you typed the data into the file,
at the end of the first three
records, you hit the enter key, to add a carriage return and line feed.
On the last record, you appear not to have.]

So, when reading the data, there is nothing after the last digit of the last value to tell parseInt() to stop reading. So, it is sitting around, hoping that more data will appear in the file.

Edit the file. At the end of the last record, hit the enter key, so that there are 4 CRs and 4 LFs in the file, not just three of each.

I am using a function to read some sensors (3) every 5 seconds and log data automaticly into an sd card file.

I am making a string containing time and the sensors readings.

Isn't File.println() supposed to do that? or should i add something to the string end?

void logData() {
  File dataFile = SD.open(filename, FILE_WRITE);
  if (dataFile) {
    String datastring = "";
    datastring = String(logtime) + ";" + String((int)(CTemp * 100)) + ";" + String((int)(CHum * 100)) + ";" + String((int)(CLight * 100));// i have just realized that my string does not have a ";" at the end
    dataFile.println(datastring);
    dataFile.close();
  }
}
14

I am making a string containing time and the sensors readings.

That is NOT what the code is doing. There is NO excuse for buffering the data in a useless String, when the SD class buffers data already.

Isn't File.println() supposed to do that?

Yes. Perhaps the issue is that the last cr/lf combo triggers an attempt to read three more values that don't exist.

That's why I never use parseInt(). I prefer to have complete control over what is happening.