Split a file.txt in multiple packages

[edward_radical]

Hi guys! I've got a file.txt which contains 30 lines. I need to iterate over the lines and divide the file in three smaller packages each with 10 lines. I was trying with this function but I know it's not correct, anyone could help me? Thanks a lot

void readHistory(){
  File file = LittleFS.open("/file.txt", "r");
  if(!file){
    Serial.println("File open failed");
  } Serial.println("------ Reading ------");

  int i = 0;
  int samples = 10;
  while (i<=samples && i!=30){
    
    String myString;
    myString.substring(i,samples); // <--- I know this is not correct but I was looking for something like this
    Serial.print("Package: ");
    Serial.println(myString);
    i = samples;
    samples = samples + i;
  }

  file.close();
}//end readHistory

[J-M-L]

1/ open your main file, if it fails, don't try to proceed.
2/ then you need to open the first output file for writing and set a count variable to 0 (check of course opening the file did work, if not stop)
3/ read a byte from the main
4/ if there is no more byte to read, then close the file, you are done.
5/ if you read a byte, write it to the output file
6/ if the byte == '\n' then increment the count variable.
7/ if count is 10 then you are done reading 10 lines. Close the output file and open the next one for writing and reset the count variable to 0
8/ go back to step 3

the DumpFile example will show you how to read data from a file. instead of writing to the serial monitor, just write to the output file.

[edward_radical]

Ok, but if I read the first 10 lines and save them in the first output file, when I open the next output file how can I start reading my main file from the 10th line?
Could you provide a little bit of code?

[J-M-L]

You don’t close the main file, just the temporary output one, so the following read in the main file will be at the start of the 11th line

I’m on my mobile so hard to put code

The examples on file dump and file read/write should give you all the function calls you need to know about

[edward_radical]

I tried this but it doesn't work

void readMain(){

  int count = 0;

  //open main file
  File file = LittleFS.open("/file.txt", "r");
  if(!file){
    Serial.println("File open failed");
  } Serial.println("------ Reading Main ------");

  while(file.available()){

    //apro il primo file
    File output = LittleFS.open("/output.txt", "w");
    if(!output){
      Serial.println("File open failed");
    } else {
      
     
      String s = file.readStringUntil('\n');
      output.println(s);
      count++;

      if (count == 10){
        File output = LittleFS.open("/output.txt", "r");
        if(!output){
          Serial.println("Output open failed");
        } Serial.println("------ Reading ------");
      
        for (int i=0; i<=10; i++){
          String lettura = output.readStringUntil('\n');
          Serial.print(i);
          Serial.print(":");
          Serial.println(lettura);
        }
        
        output.close();
        count = 0;
      }
                
    }//else
    
  }//while
  
  Serial.println("Closing File");
  file.close();
  
   
}//end readMain

I'm trying to read at least the first 10 line but the output file is empty

[J-M-L]

maybe something like this (typed here, so totally untested)

void readMain() {
  const char * outputFileNames[] = {"/out1.txt", "/out2.txt", "/out3.txt"};
  const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
  byte outputIndex = 0;

  File sourceFile;
  File destinationFile;
  String aLine;
  aLine.reserve(20);

  sourceFile = LittleFS.open("/file.txt", "r");
  if (!sourceFile) {
    Serial.println(F("Error: file.txt open failed"));
  } else {
    for (byte idx = 0; idx < outputCount; idx++) {
      if (sourceFile.available() == 0) break;
      destinationFile = LittleFS.open(outputFileNames[idx], "w");
      if (!destinationFile) {
        Serial.print(F("can't open destination "));
        Serial.println(outputFileNames[idx]);
        break;
      } else {
        int lineCount = 0;
        while (sourceFile.available() && (lineCount < 10)) {
          aLine = sourceFile.readStringUntil('\n');
          destinationFile.println(aLine); // double check if the '\n' is in the String or not (--> print or println accordingly)
          lineCount++;
        }
        destinationFile.close();
      }
    } // end for
    sourceFile.close();
  }
}

[edward_radical]

Ok thanks! This work great, sorry but I'm very new here and with c++ arduino. Could you explain to me what's the purpose of

const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
byte outputIndex = 0;

thanks a lot!

[J-M-L]

this line of code
const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
calculates the number of file names in my outputFileNames array. this way I don't have to hardcode 3

The variable byte outputIndex = 0; was meant to be an index to keep track of how many files had been written (to see if the for loop went all the way or exited early) but I forgot to add that

add this line just before we close the destination file

        outputIndex = idx; // <<---- keep track of how many files got written
        destinationFile.close();

this way at the end of the function you can return a bool if all the files were split for example

bool readMain() {
  ... // the code
 return (outputIndex == outputCount);
}

[edward_radical]

Cool that works! Last question, I implemented these lines of code in a function that everytime I read an output file the program send the values through an HTTP request. Every value of out1.txt and out2.txt is send correctly, but I receive just seven values of the third output file. What could be the problem? Thanks a lot

void httpPublish(){

  const char * outputFileNames[] = {"/out1.txt", "/out2.txt", "/out3.txt"};
  const byte outputCount = sizeof outputFileNames / sizeof outputFileNames[0];
  byte outputIndex = 0;
  
  File sourceFile;
  File destinationFile;
  
  
  const size_t capacity = JSON_OBJECT_SIZE(4) + JSON_ARRAY_SIZE(30) + 30*JSON_OBJECT_SIZE(4);
  DynamicJsonDocument doc(capacity);
  DynamicJsonDocument globalDoc(capacity);
  StaticJsonDocument <capacity> localDoc;

  String aLine;
  aLine.reserve(capacity);

  for (byte idx = 0; idx < outputCount; idx++) {
      destinationFile = LittleFS.open(outputFileNames[idx], "r");
      if (!destinationFile) {
        Serial.print(F("can't open destination "));
        Serial.println(outputFileNames[idx]);
        break;
      } else {
        Serial.print("Reading: ");
        Serial.println(outputFileNames[idx]);
        //int lineCount = 0;
        while (destinationFile.available()) {
          aLine = destinationFile.readStringUntil('\n');
          DeserializationError error = deserializeJson(localDoc, aLine);
          if (!error) globalDoc.add(localDoc);  
        }//while

        JsonObject Info = doc.createNestedObject("Info");
        Info["Battery"] = battery;
        Info["ID"] = id;
        Info["Latitudine"] = latitudine;
        Info["Longitudine"] = longitudine;
    
        
        JsonArray data = doc.createNestedArray("Data"); 
        data.add(globalDoc);
    
        HTTPClient http;
        //Send request
        http.begin("http://raspi-hyperink:1880/postjdoc");
        char buffer[capacity];
        size_t n = serializeJson(doc, buffer);
        
        http.POST(buffer);
        Serial.println(buffer);
        http.end();
        destinationFile.close();
        globalDoc.clear();
        localDoc.clear();
      }
    }// end for   
}//end httpPublish

Here's my node-red:

[J-M-L]

@edward_radical

but I receive just seven values of the third output file. What could be the problem?

probably that you only had 27 entries in your original file? (if available() says the file is empty, then we stop there)

[edward_radical]

nope the original file has 30 values I just checked printing the original file

[J-M-L]

add a debug statement in case this fails (add a else clause)

@edward_radical

          if (!error) globalDoc.add(localDoc);  

Also, is your JSON large enough ?

[edward_radical]

Ok I'll try.
I set like this the capacity of my json
const size_t capacity = JSON_OBJECT_SIZE(4) + JSON_ARRAY_SIZE(30) + 30*JSON_OBJECT_SIZE(4);

[J-M-L]

how did you come up with that size (I assume you used the assistant, but did you give the right info?)

[edward_radical]

Ok now it works, probably was just an error with the declaration of the capacity

[J-M-L]

OK

use the assistant and paste a sample JSON that is representative of the longest info you can get