Working with ESP32 SPIFFS

Hicount,

when do you start googling?
You should google at least 5 minutes.

best regards Stefan

Okay thanks Stefan. I already did that but all I can find is about opening the file and closing it again. May be I'm using the wrong keywords. What would be the exact way to define my requirement in google search? I know that I'll be needing two index to keep track of scrolling through the row and then to the column.

Do I need to use iteration through multidimensional array with pointing? or something like that? :cold_sweat:

like for example:
Get_value [ (temperature pointing to row location)(load readout point to column location)] something liike this?

https://www.google.de/search?as_q=Arduino+littlefs+reading+textfile

not the first hit but this one.
The second way to ask is I have this code that does something similar.
posting the link or the sourcecode

I tried to modify it to do .....
but it does not work yet

divide the whole thing into smaller steps

step 1 get an example code that does open a text-file, read, print read line

What does this code do

The read multiple lines is already inside

if reading multiple lines works in principle
put this code to the side.

write code that separates a string that contains a typical dataline into parts.

Don't combine it with the read from text-file! Use a variable with a fixed string.
This reduces the possabilities of bugs. If you combine it with reading from the textfile
you add more places where the bug can be.

I recommend using SafeString. Standard String-variables eat up all memory over time and then cause very hard to find programm-crashes

The library SafeString offers almost the same comfort as String-vraiables but is safe to use.

best regards Stefan

I decided to first go for the snippet of how i'm going to look for the required value after it is being fetched by esp.

So, after trying few snippets, I got this...
The sketch accepts the values over serial as it is supposed to do in actual system, search for the corresponding intersecting values and prints back it on serial monitor.
To keep things simple, I predefined an array in the sketch itself. I haven't included anything yet about SPIFFS but soon I'll. Please have a look at it and please let me know your suggestions or improvisations

const byte numChars = 50;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

// variables to hold the parsed data
int weight = 0;
int temperature = 0;
int Col_pos = 0;
int Row_pos = 0;

boolean newData = false;

int a[3][3] = {

  {2,  3,  4  },
  {5,  6,  7  },
  {8,  9, 10  }

};

void recvWithStartEndMarkers() 
{
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

//============

void parseData() {      // split the data into its parts

    char * strtokIndx; // this is used by strtok() as an index

    strtokIndx = strtok(tempChars,",");      // get the first part - the string
    weight = atoi(strtokIndx); // copy it to weight

    strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
    temperature = atoi(strtokIndx);     // copy it to temperature

 }

//============

void showParsedData() {
    Serial.print("weight: ");
    Serial.println(weight);
    Serial.print("Temperature ");
    Serial.println(temperature);
    }

void find_Number()
{
 for(int i=0; i<3; i++)
 {
  if(a[0][i] == weight)
  {
    Col_pos = i;
   }
 }

 for(int x=0; x<3; x++)
 {
  if(a[x][0] == temperature)
  {
    Row_pos = x;
  }
 }
}

//============

void setup()
{
 Serial.begin(9600);
 Serial.println("Process Begin");
 Serial.println();
 Serial.println();
}

//============

void loop()
{
    recvWithStartEndMarkers();
    if (newData == true) 
    {
        strcpy(tempChars, receivedChars);
            // this temporary copy is necessary to protect the original data
            //   because strtok() used in parseData() replaces the commas with \0
        parseData();
        showParsedData();
        newData = false;
    }
    find_Number();
    if(Row_pos != 0 && Col_pos != 0){
    Serial.print("intersect value: ");
    Serial.println(a[Row_pos][Col_pos]);
    Serial.println();
    Row_pos = 0;
    Col_pos = 0;
    }
}

//============




Hicount,

pretty well step forward.

Your receiving function "recvWithStartEndMarkers"

does no index-boundary-checking.
This means if - for what reason ever - the received string is bigger than
your array of char RAM will get overwritten.
So you should add boundary-checking

I added serial.prints to better see what your code is doing

I don't remember all details what your project shall do.
From the array with just 1..10 I do not understand it.
If you choose numbers that are very easy to identify as temperature two digits
and densitity three digits it can be seen straight forward from the number of digits if the value is a temperature or a density

Next thing is to test what happens when...

  • the string is too long
  • has no the startmarker / endmarker
  • no or wrong delimiter
  • non numerical characters like "a", "b", "c"
  • a number tooo big to fit into an integer
  • additional characters behind the endcharacter

If you have done all these tests you can be pretty sure if an error occurs
it is outside the well tested part of the code
In most cases the error will be in that part of the code that you are developing right now.

All things that happen rarely. and maybe intermittant. But if your code
can give a detailed error-message saying what happened the erro wil be found much faster.
This is something which needs time while developping or needs time when strange things are happening when the device is in use.

best regards Stefan

So here is the version with some additional serial.prints

const byte numChars = 50;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

// variables to hold the parsed data
int weight = 0;
int temperature = 0;
int Col_pos = 0;
int Row_pos = 0;

boolean newData = false;

int a[3][3] = {  {2,  3,  4  }, {5,  6,  7  }, {8,  9, 10  } };

void recvWithStartEndMarkers() {
  static boolean recvInProgress = false;
  static byte ndx = 0;
  char startMarker = '<';
  char endMarker = '>';
  char rc;

  while (Serial.available() > 0 && newData == false) {
    rc = Serial.read();

    if (recvInProgress == true) {
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else {
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newData = true;
        Serial.print("received #");
        Serial.print(receivedChars);
        Serial.println("#");
      }
    }

    else if (rc == startMarker) {
      recvInProgress = true;
    }
  }
}

//============

void parseData() {      // split the data into its parts
  char * strtokIndx; // this is used by strtok() as an index

  strtokIndx = strtok(tempChars, ",");     // get the first part - the string
  weight = atoi(strtokIndx); // copy it to weight

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  temperature = atoi(strtokIndx);     // copy it to temperature
}

//============

void showParsedData() {
  Serial.print("weight: ");
  Serial.println(weight);
  Serial.print("Temperature ");
  Serial.println(temperature);
  Serial.println();
  Serial.println();
}

void find_Number() {
  for (int i = 0; i < 3; i++) {
    Serial.print("a[0][");
    Serial.print(i);
    Serial.print("]=#");
    Serial.print(a[0][i]);
    Serial.println("#");
    if (a[0][i] == weight) {
      Serial.println("found weight ");
      Serial.println(a[0][i]);
      Serial.println();
      Serial.println();
      Col_pos = i;
    }
  }
  Serial.println();

  for (int x = 0; x < 3; x++) {
    Serial.print("a[");
    Serial.print(x);
    Serial.print("][0]=#");
    Serial.print(a[x][0]);
    Serial.println("#");
    if (a[x][0] == temperature) {
      Serial.println("found temperature ");
      Serial.println(a[x][0]);
      Serial.println();
      Serial.println();
      Row_pos = x;
    }
  }
  Serial.println();

}

//============

void setup() {
  Serial.begin(115200);
  Serial.println("Process Begin");
  Serial.println();
  Serial.println();
}

//============

void loop() {
  //Serial.print("Entering loop");
  recvWithStartEndMarkers();
  if (newData == true) {
    strcpy(tempChars, receivedChars);
    // this temporary copy is necessary to protect the original data
    //   because strtok() used in parseData() replaces the commas with \0
    parseData();
    showParsedData();
    find_Number();
    newData = false;
  }
  //find_Number();
  if (Row_pos != 0 && Col_pos != 0) {
    Serial.print("intersect value: ");
    Serial.println(a[Row_pos][Col_pos]);
    Serial.println();
    Row_pos = 0;
    Col_pos = 0;
  }
}

//============

You will have big matrixes to search, and you can speed things up by adding break statements to stop when you have found an index match.

void find_Number()
{
  for (int i = 0; i < 3; i++)
  {
    if (a[0][i] == weight)
    {
      Col_pos = i;
      break;
    }
  }

  for (int x = 0; x < 3; x++)
  {
    if (a[x][0] == temperature)
    {
      Row_pos = x;
      break;
    }
  }
}

There will be additional methods to optimize your search, but these can be worked on when you are a bit further on.

Thank you Stefan for helping with more interactive version of my code.
I have used serial communication with start and end markers in many other projects before this, it makes easy to what we can say kinda custom built protocol over UART.

To catch you up with my requirement, I'm having a Arduino UNO and NODE-MCU. My NODE-MCU will run a SPIFFS text file containing some numerical in tabulated for. My arduino has a loadcell and NTC which reads data from these two and sends it to NODE-MCU over UART using start and end markers. I'm following this one <loadcell value, NTC value> anything before and after the marker will get ignored automatically.

About the concerns you raised, I really didn't got what boundary checking would be like. (I've done couple of programming but I'm not like an expert :frowning: )
so from my earlier experience with start and end markers was that, if the received string is bigger than the capacity of the array, it either prints gibberish values on serial monitor or restarts! The solution to this problem which I adopted was not really something technically fit. I just went on increasing the array capacity until it stopped happening. It took me several iterations but after few days I got it right.

But here the case is going to be entirely different and I need something firm to decide the size of an array. After googling for it some a bit, I found people using something like this: (sizeof(array))/(sizeof(array[0]) will it work in my case? I'm supposed to get all the data from SPIFFS text into the array and then perform the above operation for which I have written the code rn. This is where I would like to have your valuable suggestions

Unfortunately, I do not have NODE-MCU rn and we are in lockdown, so I can't go out and get it, even the local transports are closed here. My efforts are atleast getting a rough sketch by the time the lockdown lifts up and I can get my NODE-MCU.

Good thing about the text file is that I won't be facing any alphabets or non-numerical characters!
Once again thanks for more improved version of my code, I'll try it out and post what happens

Regards Count

Thanks Cattledog! I'll make those changes. :slight_smile:

Hicount,

if you define an array

const byte numChars = 50;
char receivedChars[numChars];

you can store chars from index 0 to index 49
receivedChars[0] = 'a';
receivedChars[1] = 'b';
...
receivedChars[49] = 'Z'; // or better the terminating zero
receivedChars[49] = 0;

if you do
receivedChars[50] = 'Y';

this makes c++ write a byte into RAM at an adress that is used for something else
because the RAM-area for your arrays of chars ends at index 49

you have to check if your variable ndx wants to become greater than 49

I have taken a closer look at your code and dicovered you do boundary checking

if (ndx >= numChars) {
          ndx = numChars - 1;
        }

sorry that I have overlooked this.

Using the PString or the SafeString-libary does boundary-checking "under the hood" internal. The effect that can be seen "on the surface"
is that a too long string gets truncated. But trying to write beyond the boundary is not executed and the bytes you tried to write behind the boundary just get dropped. And this has the effect that the code is not crashing.

How time do you have between reading load-cell and temperature until the resulting densitiy must be delivered?

is it timecritical = must be within 0,01 seconds or do you have more time?

If it is timecritical the searching through all the values must be optimised.
If you have mutliple seconds you can use hundreds of files with each hundreds of lines to read in sequentually. And this will still take just one or two seconds - I estimate.

best regards Stefan

One way you will be able to speed things up is if all the data values for the different temperatures at a given weight create a file of the same size.
That way when you know the index of the weight, you can use a known offset to seek the line to parse for the value at a temperature. Otherwise you will have to read lines to an end marker and count lines.

For example if you have 10 lines(index 0-9) for different weights and 10 values for "density" at a 10 different temperatures(index 0-9) and all the density values and their separators add up to 200 bytes you can quickly find line 5(index4) with file.seek(4*200) instead of having to read each line to count end markers.

So the answer to your question can be quite confusing. The guy for whom I'm doing this job have asked me to make two modes in the system.
Mode 1: Where the readings from loadcell and NTC will be fetched once and density will be showed.
Mode 2: Which he calls "live" mode where the system continuously keeps on reading the NTC (No need to check load cell just the NTC) and makes changes in the density value accordingly.
Did I made it simple enough? or still confusing?

So, I guess, for the Mode 1 I don't see any time criticalness. But for Mode2 it can be time critical as I need to keep reading the NTC and reflect the changes accordingly.

Yes! that's what the earlier developer did. I'm not able to understand it all.

Cattledog, it will be a huge favor, if u could explain it in bit more detail to help me understand it. The developer who was working on this before me used to get the load cell value and run some calculations on it and Bingo! there he finds which file to refer! he created 14 different text files and each text file was named according to the weights. Once he finds the file he just reads the NTC and runs some calculations on it as well and there he finds the density value as well!

particular thing to note about this was I went through the text files he created and I didn't find any temperature values written there in any of its 14 files!

So how does this exactly works? If you want I can share the snippet or entire code from the previous developer.

You posted an part of an excel-sheet with temperatures and densitiy-values.

I mean this one
image

Do you have all and every temperature-density combinations in this excel-sheet?

As the former developper did not document how his code works it might take 50 hours to anaylse it.
It will be much faster to use the excel-sheets data as a starting point and write the code from scratch.

From this excel-sheet you can create a textfile with comma-seperated values.

This textfile can be stored into SPIFFS. If this textfiles has 3 Megabytes or more you would need to add an SD-card to read it from the SD-card.

About timecritical or not.
continous-reading is not a sufficient description of the requierements

continous-reading can mean read ntc-temeparure once per minute do a reading and do this once-per-minute-reading for hours and just show some values just for information.

it could mean read ntc-temeparure every 0,1 seconds and do additional calculations that are summarising values and an error in too late reading would add up a big error in the calculations over time.

So do you have a detailed description of what the system should do?

Reading some hundred lines of a textfile will be done in one second
so if "continious reading" means upfdate value every 2 or 3 seconds will be sufficient
just crawling throw all lines of the textfile will be sufficient.

You are working on an informatic project. And what is mostly needed in an informatic project is information

and a very good way to understand what shall going on is describing an EXAMPLE with NUMBERS that show the whole process.

If you can't describe it in this way how should other user here see in the glas-sphere how the programming could look like?????!

best regards Stefan

You appeared to be going down a path where you found two index values. One for the weight and one for the temperature. These two indexes referenced a "density" value. You say the density values are text representations of floating point numbers("xxx.x").

I would also consider multiplying by 10 and using a uint16_t integer for them as it will make the data file smaller. The data appears to be constant and unchanging in the program which eliminates dynamic memory issues.

It's not clear to me yet about the number of weights and temperatures in this array and what its size in bytes actually is. Providing the excel spreadsheet for us to look at would be good.

You may indeed be able to hold the values in an integer array myData[][], keep it in ram and access by index. A 2d array bears close relation to a multi-line csv text file, and that is another way to hold the data, and if the file very large, using SPIFFS and a file management approach can be used.

I'm not clear if there is an advantage to having multiple single line files named by weight, or a single multi line file in which you access the line by the index of the weight. In the single file, finding the correct line for example weight 789 which might be index 123 is the issue. You can either start at the beginning, read lines until an end marker and count until you find line 123 and then look for the density. Alternatively, if the line are all the same size, you don't have to read each one but you can find the index by offset from the start. I have no direct experience with the speed of reading through hundreds of lines to find an end marker to count, but Stefan says it can be pretty fast. Still, I think that having all the lines be the same length is easier if you can indeed get the data into that form.

If you have multiple files named by weight, you do not need to do the index finding for weight but you will need convert the number to a name.

I agree with Stefan, the more specific and detailed information you provide, the more help you can receive.

So I'll try to answer all the questions step by step in the best possible way I can...

  1. Do I have all the temp-density combinations in my excel sheet?
    Ans: I do have an excel sheet containing all the values and its combination.

  2. Continuous reading for NTC
    Ans: By Continuous reading I meant, The UNO sends NTC reading to ESP, ESP sorts out the temp-weight-density combination, reverts the density value back to UNO and the UNO displays it on LCD. UNO then again fetches the NTC value and cycle repeats.

  3. How it is expected to work when I say not continuous.
    Ans: Fetching the NTC value only once would be enough

although these two features can be ignored for now. I'll try to discuss with that guy and see what more about this I can have from him.

Meanwhile, I'm really concerned about the time the NODE MCU will take to search for the values. Does it mean that the code that I wrote is totally useless? :frowning:

No the opposite is the case. It is a good base if all the temperature and densitiy values is stored in one or multiple textfiles.
Reading in a line from a textfile and then your code does extract the values. This is fine.

I haven't done speed-measurings how long it takes to read in a tectline and how long it takes to read in all lines until the right values were found.
My rough expectation is it will only take 1 or 2 seconds.

You have described the process:

But this description contains no information about how long a single cycle is allowed to be.
From what you have described so far I assume the displayed value is for
somebody is looking onto the display just to see the value or maybe to write it down on a sheet of paper and thats all
and that the temperarute hence the density is changing slow enough that it doesn't matter to read the new value and displaying this value two seconds later.

The design to have the values dispensed over multiple files is not nescessary.

You can have the values in one file.
The difference is just
choose right filename or choose right columm if all values are collected in a single file.

You should think about the pros and cons of both possabilities and make a decision, because the code will look different in some parts depending on the chosen way to organise the temperature/density data
best regards Stefan

You should be able to convert the excel file to .pdf and attach it.

The UNO sends NTC reading to ESP, ESP sorts out the temp-weight-density combination, reverts the density value back to UNO and the UNO displays it on LCD.

Question: Why do you need the UNO to read the temperature and run the display. Can't the entire project be done on the ESP32?

Exactly, it really doesn't matter how much time it takes to show the result until its not in minutes. Few seconds is really fine and it won't even matter.

Its becox the hardware in use rn was designed for some other use and now the guy wants the hardware remains same but runs the other sketch on it. once the sketch works okay on this later a dedicated system will be designd for this.