How to minimize the size of a json file

davidmarli · August 17, 2024, 6:39am

Hi,

for my project I need to store datas in a json file in littlefs partition.
The json file contains 140 strings which are all the same type : "XX:XX" (time hour minute),
it contains also 42 bools
When I want to load and read this file, I declare

String StartHour[70];
String EndHour[70];
bool CheckTempo[42];

To read and load the value, I use

fs::File configFile = LittleFS.open(filename, "r");
  StaticJsonDocument<4096> doc;
  DeserializationError error = deserializeJson(doc, configFile);
  if (error) {
    Serial.println(F("Failed to read file, using default configuration in function loadConfiguration"));
  }
configrelais.CheckJoursTEMPO[0] = doc["CheckJoursRouge_Lun1"].as<bool>();
 configrelais.CheckJoursTEMPO[1] = doc["CheckJoursBlanc_Lun1"].as<bool>();
 configrelais.CheckJoursTEMPO[2] = doc["CheckJoursBleu_Lun1"].as<bool>();
 configrelais.CheckJoursTEMPO[3] = doc["CheckJoursRouge_Mar1"].as<bool>();
 configrelais.CheckJoursTEMPO[4] = doc["CheckJoursBlanc_Mar1"].as<bool>();
 configrelais.CheckJoursTEMPO[5] = doc["CheckJoursBleu_Mar1"].as<bool>();
 configrelais.CheckJoursTEMPO[6] = doc["CheckJoursRouge_Mer1"].as<bool>();
 configrelais.CheckJoursTEMPO[7] = doc["CheckJoursBlanc_Mer1"].as<bool>();
 configrelais.CheckJoursTEMPO[8] = doc["CheckJoursBleu_Mer1"].as<bool>();
 configrelais.CheckJoursTEMPO[9] = doc["CheckJoursRouge_Jeu1"].as<bool>();
 configrelais.CheckJoursTEMPO[10] = doc["CheckJoursBlanc_Jeu1"].as<bool>();
 configrelais.CheckJoursTEMPO[11] = doc["CheckJoursBleu_Jeu1"].as<bool>();
 configrelais.CheckJoursTEMPO[12] = doc["CheckJoursRouge_Ven1"].as<bool>();
 configrelais.CheckJoursTEMPO[13] = doc["CheckJoursBlanc_Ven1"].as<bool>();
 configrelais.CheckJoursTEMPO[14] = doc["CheckJoursBleu_Ven1"].as<bool>();
 configrelais.CheckJoursTEMPO[15] = doc["CheckJoursRouge_Sam1"].as<bool>();
 configrelais.CheckJoursTEMPO[16] = doc["CheckJoursBlanc_Sam1"].as<bool>();
 configrelais.CheckJoursTEMPO[17] = doc["CheckJoursBleu_Sam1"].as<bool>();
 configrelais.CheckJoursTEMPO[18] = doc["CheckJoursRouge_Dim1"].as<bool>();
 configrelais.CheckJoursTEMPO[19] = doc["CheckJoursBlanc_Dim1"].as<bool>();
 configrelais.CheckJoursTEMPO[20] = doc["CheckJoursBleu_Dim1"].as<bool>();

//RELAIS 2
 configrelais.CheckJoursTEMPO[0+21] = doc["CheckJoursRouge_Lun2"].as<bool>();
 configrelais.CheckJoursTEMPO[1+21] = doc["CheckJoursBlanc_Lun2"].as<bool>();
 configrelais.CheckJoursTEMPO[2+21] = doc["CheckJoursBleu_Lun2"].as<bool>();
 configrelais.CheckJoursTEMPO[3+21] = doc["CheckJoursRouge_Mar2"].as<bool>();
 configrelais.CheckJoursTEMPO[4+21] = doc["CheckJoursBlanc_Mar2"].as<bool>();
 configrelais.CheckJoursTEMPO[5+21] = doc["CheckJoursBleu_Mar2"].as<bool>();
 configrelais.CheckJoursTEMPO[6+21] = doc["CheckJoursRouge_Mer2"].as<bool>();
 configrelais.CheckJoursTEMPO[7+21] = doc["CheckJoursBlanc_Mer2"].as<bool>();
 configrelais.CheckJoursTEMPO[8+21] = doc["CheckJoursBleu_Mer2"].as<bool>();
 configrelais.CheckJoursTEMPO[9+21] = doc["CheckJoursRouge_Jeu2"].as<bool>();
 configrelais.CheckJoursTEMPO[10+21] = doc["CheckJoursBlanc_Jeu2"].as<bool>();
 configrelais.CheckJoursTEMPO[11+21] = doc["CheckJoursBleu_Jeu2"].as<bool>();
 configrelais.CheckJoursTEMPO[12+21] = doc["CheckJoursRouge_Ven2"].as<bool>();
 configrelais.CheckJoursTEMPO[13+21] = doc["CheckJoursBlanc_Ven2"].as<bool>();
 configrelais.CheckJoursTEMPO[14+21] = doc["CheckJoursBleu_Ven2"].as<bool>();
 configrelais.CheckJoursTEMPO[15+21] = doc["CheckJoursRouge_Sam2"].as<bool>();
 configrelais.CheckJoursTEMPO[16+21] = doc["CheckJoursBlanc_Sam2"].as<bool>();
 configrelais.CheckJoursTEMPO[17+21] = doc["CheckJoursBleu_Sam2"].as<bool>();
 configrelais.CheckJoursTEMPO[18+21] = doc["CheckJoursRouge_Dim2"].as<bool>();
 configrelais.CheckJoursTEMPO[19+21] = doc["CheckJoursBlanc_Dim2"].as<bool>();
 configrelais.CheckJoursTEMPO[20+21] = doc["CheckJoursBleu_Dim2"].as<bool>();

//PLAGE 1 RELAIS1
configrelais.StartHour[0] = doc["H1Start_Lun1"].as<String>();
configrelais.EndHour[0] = doc["H1End_Lun1"].as<String>();
configrelais.StartHour[1] = doc["H1Start_Mar1"].as<String>();
configrelais.EndHour[1] = doc["H1End_Mar1"].as<String>();
configrelais.StartHour[2] = doc["H1Start_Mer1"].as<String>();
configrelais.EndHour[2] = doc["H1End_Mer1"].as<String>();
configrelais.StartHour[3] = doc["H1Start_Jeu1"].as<String>();
configrelais.EndHour[3] = doc["H1End_Jeu1"].as<String>();
configrelais.StartHour[4] = doc["H1Start_Ven1"].as<String>();
configrelais.EndHour[4] = doc["H1End_Ven1"].as<String>();
configrelais.StartHour[5] = doc["H1Start_Sam1"].as<String>();
configrelais.EndHour[5] = doc["H1End_Sam1"].as<String>();
configrelais.StartHour[6] = doc["H1Start_Dim1"].as<String>();
configrelais.EndHour[6] = doc["H1End_Dim1"].as<String>();

//PLAGE 2 RELAIS1
configrelais.StartHour[0+7] = doc["H2Start_Lun1"].as<String>();
configrelais.EndHour[0+7] = doc["H2End_Lun1"].as<String>();
configrelais.StartHour[1+7] = doc["H2Start_Mar1"].as<String>();
configrelais.EndHour[1+7] = doc["H2End_Mar1"].as<String>();
configrelais.StartHour[2+7] = doc["H2Start_Mer1"].as<String>();
configrelais.EndHour[2+7] = doc["H2End_Mer1"].as<String>();
configrelais.StartHour[3+7] = doc["H2Start_Jeu1"].as<String>();
configrelais.EndHour[3+7] = doc["H2End_Jeu1"].as<String>();
configrelais.StartHour[4+7] = doc["H2Start_Ven1"].as<String>();
configrelais.EndHour[4+7] = doc["H2End_Ven1"].as<String>();
configrelais.StartHour[5+7] = doc["H2Start_Sam1"].as<String>();
configrelais.EndHour[5+7] = doc["H2End_Sam1"].as<String>();
configrelais.StartHour[6+7] = doc["H2Start_Dim1"].as<String>();
configrelais.EndHour[6+7] = doc["H2End_Dim1"].as<String>();

//PLAGE 3 RELAIS1
configrelais.StartHour[0+14] = doc["H3Start_Lun1"].as<String>();
configrelais.EndHour[0+14] = doc["H3End_Lun1"].as<String>();
configrelais.StartHour[1+14] = doc["H3Start_Mar1"].as<String>();
configrelais.EndHour[1+14] = doc["H3End_Mar1"].as<String>();
configrelais.StartHour[2+14] = doc["H3Start_Mer1"].as<String>();
configrelais.EndHour[2+14] = doc["H3End_Mer1"].as<String>();
configrelais.StartHour[3+14] = doc["H3Start_Jeu1"].as<String>();
configrelais.EndHour[3+14] = doc["H3End_Jeu1"].as<String>();
configrelais.StartHour[4+14] = doc["H3Start_Ven1"].as<String>();
configrelais.EndHour[4+14] = doc["H3End_Ven1"].as<String>();
configrelais.StartHour[5+14] = doc["H3Start_Sam1"].as<String>();
configrelais.EndHour[5+14] = doc["H3End_Sam1"].as<String>();
configrelais.StartHour[6+14] = doc["H3Start_Dim1"].as<String>();
configrelais.EndHour[6+14] = doc["H3End_Dim1"].as<String>();

//PLAGE 4 RELAIS1
configrelais.StartHour[0+21] = doc["H4Start_Lun1"].as<String>();
configrelais.EndHour[0+21] = doc["H4End_Lun1"].as<String>();
configrelais.StartHour[1+21] = doc["H4Start_Mar1"].as<String>();
configrelais.EndHour[1+21] = doc["H4End_Mar1"].as<String>();
configrelais.StartHour[2+21] = doc["H4Start_Mer1"].as<String>();
configrelais.EndHour[2+21] = doc["H4End_Mer1"].as<String>();
configrelais.StartHour[3+21] = doc["H4Start_Jeu1"].as<String>();
configrelais.EndHour[3+21] = doc["H4End_Jeu1"].as<String>();
configrelais.StartHour[4+21] = doc["H4Start_Ven1"].as<String>();
configrelais.EndHour[4+21] = doc["H4End_Ven1"].as<String>();
configrelais.StartHour[5+21] = doc["H4Start_Sam1"].as<String>();
configrelais.EndHour[5+21] = doc["H4End_Sam1"].as<String>();
configrelais.StartHour[6+21] = doc["H4Start_Dim1"].as<String>();
configrelais.EndHour[6+21] = doc["H4End_Dim1"].as<String>();

//PLAGE 5 RELAIS1
configrelais.StartHour[0+28] = doc["H5Start_Lun1"].as<String>();
configrelais.EndHour[0+28] = doc["H5End_Lun1"].as<String>();
configrelais.StartHour[1+28] = doc["H5Start_Mar1"].as<String>();
configrelais.EndHour[1+28] = doc["H5End_Mar1"].as<String>();
configrelais.StartHour[2+28] = doc["H5Start_Mer1"].as<String>();
configrelais.EndHour[2+28] = doc["H5End_Mer1"].as<String>();
configrelais.StartHour[3+28] = doc["H5Start_Jeu1"].as<String>();
configrelais.EndHour[3+28] = doc["H5End_Jeu1"].as<String>();
configrelais.StartHour[4+28] = doc["H5Start_Ven1"].as<String>();
configrelais.EndHour[4+28] = doc["H5End_Ven1"].as<String>();
configrelais.StartHour[5+28] = doc["H5Start_Sam1"].as<String>();
configrelais.EndHour[5+28] = doc["H5End_Sam1"].as<String>();
configrelais.StartHour[6+28] = doc["H5Start_Dim1"].as<String>();
configrelais.EndHour[6+28] = doc["H5End_Dim1"].as<String>();

//PLAGE 1 RELAIS2
configrelais.StartHour[0+35] = doc["H1Start_Lun2"].as<String>();
Serial.println(doc["H1Start_Lun2"].as<String>());
configrelais.EndHour[0+35] = doc["H1End_Lun2"].as<String>();
configrelais.StartHour[1+35] = doc["H1Start_Mar2"].as<String>();
configrelais.EndHour[1+35] = doc["H1End_Mar2"].as<String>();
configrelais.StartHour[2+35] = doc["H1Start_Mer2"].as<String>();
configrelais.EndHour[2+35] = doc["H1End_Mer2"].as<String>();
configrelais.StartHour[3+35] = doc["H1Start_Jeu2"].as<String>();
configrelais.EndHour[3+35] = doc["H1End_Jeu2"].as<String>();
configrelais.StartHour[4+35] = doc["H1Start_Ven2"].as<String>();
configrelais.EndHour[4+35] = doc["H1End_Ven2"].as<String>();
configrelais.StartHour[5+35] = doc["H1Start_Sam2"].as<String>();
configrelais.EndHour[5+35] = doc["H1End_Sam2"].as<String>();
configrelais.StartHour[6+35] = doc["H1Start_Dim2"].as<String>();
configrelais.EndHour[6+35] = doc["H1End_Dim2"].as<String>();

//PLAGE 2 RELAIS2
configrelais.StartHour[0+42] = doc["H2Start_Lun2"].as<String>();
Serial.println(doc["H2Start_Lun2"].as<String>());
configrelais.EndHour[0+42] = doc["H2End_Lun2"].as<String>();
configrelais.StartHour[1+42] = doc["H2Start_Mar2"].as<String>();
configrelais.EndHour[1+42] = doc["H2End_Mar2"].as<String>();
configrelais.StartHour[2+42] = doc["H2Start_Mer2"].as<String>();
configrelais.EndHour[2+42] = doc["H2End_Mer2"].as<String>();
configrelais.StartHour[3+42] = doc["H2Start_Jeu2"].as<String>();
configrelais.EndHour[3+42] = doc["H2End_Jeu2"].as<String>();
configrelais.StartHour[4+42] = doc["H2Start_Ven2"].as<String>();
configrelais.EndHour[4+42] = doc["H2End_Ven2"].as<String>();
configrelais.StartHour[5+42] = doc["H2Start_Sam2"].as<String>();
configrelais.EndHour[5+42] = doc["H2End_Sam2"].as<String>();
configrelais.StartHour[6+42] = doc["H2Start_Dim2"].as<String>();
configrelais.EndHour[6+42] = doc["H2End_Dim2"].as<String>();

//PLAGE 3 RELAIS2
configrelais.StartHour[0+49] = doc["H3Start_Lun2"].as<String>();
Serial.println(doc["H3Start_Lun2"].as<String>());
configrelais.EndHour[0+49] = doc["H3End_Lun2"].as<String>();
configrelais.StartHour[1+49] = doc["H3Start_Mar2"].as<String>();
configrelais.EndHour[1+49] = doc["H3End_Mar2"].as<String>();
configrelais.StartHour[2+49] = doc["H3Start_Mer2"].as<String>();
configrelais.EndHour[2+49] = doc["H3End_Mer2"].as<String>();
configrelais.StartHour[3+49] = doc["H3Start_Jeu2"].as<String>();
configrelais.EndHour[3+49] = doc["H3End_Jeu2"].as<String>();
configrelais.StartHour[4+49] = doc["H3Start_Ven2"].as<String>();
configrelais.EndHour[4+49] = doc["H3End_Ven2"].as<String>();
configrelais.StartHour[5+49] = doc["H3Start_Sam2"].as<String>();
configrelais.EndHour[5+49] = doc["H3End_Sam2"].as<String>();
configrelais.StartHour[6+49] = doc["H3Start_Dim2"].as<String>();
configrelais.EndHour[6+49] = doc["H3End_Dim2"].as<String>();

//PLAGE 4 RELAIS2
configrelais.StartHour[0+56] = doc["H4Start_Lun2"].as<String>();
Serial.println(doc["H4Start_Lun2"].as<String>());
configrelais.EndHour[0+56] = doc["H4End_Lun2"].as<String>();
configrelais.StartHour[1+56] = doc["H4Start_Mar2"].as<String>();
configrelais.EndHour[1+56] = doc["H4End_Mar2"].as<String>();
configrelais.StartHour[2+56] = doc["H4Start_Mer2"].as<String>();
configrelais.EndHour[2+56] = doc["H4End_Mer2"].as<String>();
configrelais.StartHour[3+56] = doc["H4Start_Jeu2"].as<String>();
configrelais.EndHour[3+56] = doc["H4End_Jeu2"].as<String>();
configrelais.StartHour[4+56] = doc["H4Start_Ven2"].as<String>();
configrelais.EndHour[4+56] = doc["H4End_Ven2"].as<String>();
configrelais.StartHour[5+56] = doc["H4Start_Sam2"].as<String>();
configrelais.EndHour[5+56] = doc["H4End_Sam2"].as<String>();
configrelais.StartHour[6+56] = doc["H4Start_Dim2"].as<String>();
configrelais.EndHour[6+56] = doc["H4End_Dim2"].as<String>();

//PLAGE 5 RELAIS2
configrelais.StartHour[0+63] = doc["H5Start_Lun2"].as<String>();
Serial.println(doc["H5Start_Lun2"].as<String>());
configrelais.EndHour[0+63] = doc["H5End_Lun2"].as<String>();
configrelais.StartHour[1+63] = doc["H5Start_Mar2"].as<String>();
configrelais.EndHour[1+63] = doc["H5End_Mar2"].as<String>();
configrelais.StartHour[2+63] = doc["H5Start_Mer2"].as<String>();
configrelais.EndHour[2+63] = doc["H5End_Mer2"].as<String>();
configrelais.StartHour[3+63] = doc["H5Start_Jeu2"].as<String>();
configrelais.EndHour[3+63] = doc["H5End_Jeu2"].as<String>();
configrelais.StartHour[4+63] = doc["H5Start_Ven2"].as<String>();
configrelais.EndHour[4+63] = doc["H5End_Ven2"].as<String>();
configrelais.StartHour[5+63] = doc["H5Start_Sam2"].as<String>();
configrelais.EndHour[5+63] = doc["H5End_Sam2"].as<String>();
configrelais.StartHour[6+63] = doc["H5Start_Dim2"].as<String>();
configrelais.EndHour[6+63] = doc["H5End_Dim2"].as<String>();
configFile.close();

The first values are ok, and then it becomes "null".

I think it's a memory size with StaticJsonDocument<4096> doc;

So, first when I declare the variables, is there a way to tell that my string are all 5 caracters ?
String[5] StartHour[70] : but that's not correct when I compil.

Thank you

UKHeliBob · August 17, 2024, 6:43am

Why not use C style strings instead of Strings ?
Do you actually need the colon separator in the hours/minutes values ?

davidmarli · August 17, 2024, 6:53am

Thank You

I don't what is C style strings ...
Yes I need the separator.

noiasca · August 17, 2024, 7:04am

sounds weird to store an hour in a String Object...

upload the full JSON and tell, which ArduinoJSON library version you are using.

davidmarli · August 17, 2024, 7:06am

{  
    "CheckJoursRouge_Lun1" : false,
    "CheckJoursBlanc_Lun1" : false,
    "CheckJoursBleu_Lun1" : false,
    "CheckJoursRouge_Mar1" : false,
    "CheckJoursBlanc_Mar1" : false,
    "CheckJoursBleu_Mar1" : false,
    "CheckJoursRouge_Mer1" : false,
    "CheckJoursBlanc_Mer1" : false,
    "CheckJoursBleu_Mer1" : false,
    "CheckJoursRouge_Jeu1" :false,
    "CheckJoursBlanc_Jeu1" : false,
    "CheckJoursBleu_Jeu1" : false,
    "CheckJoursRouge_Ven1" : false,
    "CheckJoursBlanc_Ven1" : false,
    "CheckJoursBleu_Ven1" : false,
    "CheckJoursRouge_Sam1" : false,
    "CheckJoursBlanc_Sam1" : false,
    "CheckJoursBleu_Sam1" : false,
    "CheckJoursRouge_Dim1" : false,
    "CheckJoursBlanc_Dim1" : false,
    "CheckJoursBleu_Dim1" : false,
    
    "H1Start_Lun1"  : "06:00",	
    "H1End_Lun1"  : "06:53",
    "H1Start_Mar1"  : "99:99",
    "H1End_Mar1"  : "99:99",
    "H1Start_Mer1"  : "99:99",
     "H1End_Mer1"  : "99:99",
     "H1Start_Jeu1"  : "99:99",
     "H1End_Jeu1"  : "99:99",
     "H1Start_Ven1"  : "99:99",
     "H1End_Ven1"  : "99:99",
     "H1Start_Sam1"  : "99:99",
     "H1End_Sam1"  : "99:99",
     "H1Start_Dim1"  : "99:99",
     "H1End_Dim1"  : "99:99",

     "H2Start_Lun1"  : "99:99",
     "H2End_Lun1"  : "99:99",
     "H2Start_Mar1"  : "07:00",
     "H2End_Mar1"  : "07:54",
     "H2Start_Mer1"  : "99:99",
     "H2End_Mer1"  : "99:99",
     "H2Start_Jeu1"  : "99:99",
     "H2End_Jeu1"  : "99:99",
     "H2Start_Ven1"  : "99:99",
     "H2End_Ven1"  : "99:99",
     "H2Start_Sam1"  : "99:99",
     "H2End_Sam1"  : "99:99",
     "H2Start_Dim1"  : "99:99",
     "H2End_Dim1"  : "99:99",

     "H3Start_Lun1"  : "99:99",
     "H3End_Lun1"  : "99:99",
     "H3Start_Mar1"  : "99:99",
     "H3End_Mar1"  : "99:99",
     "H3Start_Mer1"  : "99:99",
     "H3End_Mer1"  : "99:99",
     "H3Start_Jeu1"  : "99:99",
     "H3End_Jeu1"  : "99:99",
     "H3Start_Ven1"  : "99:99",
     "H3End_Ven1"  : "99:99",
     "H3Start_Sam1"  : "99:99",
     "H3End_Sam1"  : "99:99",
     "H3Start_Dim1"  : "99:99",
     "H3End_Dim1"  : "99:99",

     "H4Start_Lun1"  : "99:99",
     "H4End_Lun1"  : "99:99",
     "H4Start_Mar1"  : "99:99",
     "H4End_Mar1"  : "99:99",
     "H4Start_Mer1"  : "99:99",
     "H4End_Mer1"  : "99:99",
     "H4Start_Jeu1"  : "99:99",
     "H4End_Jeu1"  : "99:99",
     "H4Start_Ven1"  : "99:99",
     "H4End_Ven1"  : "99:99",
     "H4Start_Sam1"  : "99:99",
     "H4End_Sam1"  : "99:99",
     "H4Start_Dim1"  : "99:99",
     "H4End_Dim1"  : "99:99",

     "H5Start_Lun1"  : "99:99",
     "H5End_Lun1"  : "99:99",
     "H5Start_Mar1"  : "99:99",
     "H5End_Mar1"  : "99:99",
     "H5Start_Mer1"  : "99:99",
     "H5End_Mer1"  : "99:99",
     "H5Start_Jeu1"  : "99:99",
     "H5End_Jeu1"  : "99:99",
     "H5Start_Ven1"  : "99:99",
     "H5End_Ven1"  : "99:99",
     "H5Start_Sam1"  : "99:99",
     "H5End_Sam1"  : "99:99",
     "H5Start_Dim1"  : "99:99",
     "H5End_Dim1"  : "99:99"
        
    
}

"99:99" is needed so that I can know this time is not used.
Librairy : bblanchon/ArduinoJson@^6.17.3

Json file is not complete. I just psate half of it.
For know I store data in two json file. I can allow 4096 for jsondocument. And it goes well.

But programmaing is just an hobby and certainly not my job, I'm pretty sure it can be better

sterretje · August 17, 2024, 7:07am

char someText[6];

The above is a char array that can hold 5 characters and a terminating '\0'. And below is an array of char arrays for 20 texts.

char someTextArray[20][6];

I don't know much about json and I don't know anything about ESP boards.

Maybe the following will get you on the way:

Change CheckJoursRouge_Lun1 to ChkJoursRouge_Lun1; it saves two characters in the json file; repeat for others.
Change H1Start_Lun1 to H1S_Lun1; 4 characters saved in the json file; repeat for others.

Have you tried to make it bigger? E.g.

StaticJsonDocument<8192> doc;

davidmarli · August 17, 2024, 7:10am

Thank you :

char someTextArray[20][6];

Is what i didn't know.

Shrink the name is a good idea.

<8192> make the ESP crash because it has not enough memory space.

noiasca · August 17, 2024, 7:45am

then it is useless for us and you have to use the ArduinoJSON Assistant on your own.

Furthermore you should think about your general data structure.

qualifies like

CheckJoursRouge_Lun1

and all its color-weekday-Number variants are space wasting when you really need to minimize the JSON file size.

it seems you have three colors for the 7 weekdays - each 2 times
r (red)
w (white)
b (blue)
an array for the 7 weekdays
and then two bool values one for xx1 and one for xx2

As alternative you can even combine the start end times in your datastructure and start with the weekday (as array)

Giving the data a well defined structure will also reduce the lines of code.
But your target was to minimize the JSON file size. And this could be accomplished by cleaning up the JSON itself.

rsmls · August 17, 2024, 8:10am

Where does this requirement stem from?

You appear to be dealing with data with a static structure; i.e. all data have a fixed size, there is a fixed number of elements and they appear in a fixed order. You might as well store them in a regular text file and encode in whatever way is the smallest. I'd start with removing the lengthy element names since the vast majority of your json file is made up of these, while they ultimately end up being stored in an array within some kind of object in your sketch.

Does some other device/process access the same json file? Is there a possibility of having that process/device handle a different kind of file format?

Json is of course great because of its interoperability. But it was never optimized for small size in particular. So if the latter is more relevant than the former, json may not be the most suitable approach.

davidmarli · August 17, 2024, 8:28am

I'm the only unsing this json file.

All datas have the same size (XX:XX), except the 3 first one (an ip-adress, a boolean ant an integer)
Yes I can store all datas in a text file.

But then How do I retrieve datas ?
I just save datas in a certain order, and then read them the same way line by line and affect them to the right place in my arrays ?

noiasca · August 17, 2024, 9:20am

XX:XX do you need to store the : ?
why not an integer in HHMM?

We don't know enough about your project, where this JSON comes from or why it needs to be stored in littlefs.
We don't know which microcontroller you are using.
Or why you don't use the preferences.
Or why you don't just hold your settings in a data structure and store the binary data of the data structure.

J-M-L · August 17, 2024, 9:30am

I would create structures that are the exact replica of your data

struct __attribute__((packed)) HourMinute {
  byte hour;
  byte minute;
};

struct __attribute__((packed)) Configuration {
  bool flags[42];
  HourMinute timing[140];
};

you could have a default structure in your code with relevant values

Configuration config = {
  {
    true, true, false, false, true, true, false, false, true, true,
    false, false, true, true, false, false, true, true, false, false,
    true, true, false, false, true, true, false, false, true, true,
    false, false, true, true, false, false, true, true, false, false,
    true, true
  },
  {
    { 0,  0}, { 0,  5}, { 0, 10}, { 0, 15}, { 0, 20}, { 0, 25}, { 0, 30}, { 0, 35}, { 0, 40}, { 0, 45},
    { 0, 50}, { 0, 55}, { 1,  0}, { 1,  5}, { 1, 10}, { 1, 15}, { 1, 20}, { 1, 25}, { 1, 30}, { 1, 35},
    { 1, 40}, { 1, 45}, { 1, 50}, { 1, 55}, { 2,  0}, { 2,  5}, { 2, 10}, { 2, 15}, { 2, 20}, { 2, 25},
    { 2, 30}, { 2, 35}, { 2, 40}, { 2, 45}, { 2, 50}, { 2, 55}, { 3,  0}, { 3,  5}, { 3, 10}, { 3, 15},
    { 3, 20}, { 3, 25}, { 3, 30}, { 3, 35}, { 3, 40}, { 3, 45}, { 3, 50}, { 3, 55}, { 4,  0}, { 4,  5},
    { 4, 10}, { 4, 15}, { 4, 20}, { 4, 25}, { 4, 30}, { 4, 35}, { 4, 40}, { 4, 45}, { 4, 50}, { 4, 55},
    { 5,  0}, { 5,  5}, { 5, 10}, { 5, 15}, { 5, 20}, { 5, 25}, { 5, 30}, { 5, 35}, { 5, 40}, { 5, 45},
    { 5, 50}, { 5, 55}, { 6,  0}, { 6,  5}, { 6, 10}, { 6, 15}, { 6, 20}, { 6, 25}, { 6, 30}, { 6, 35},
    { 6, 40}, { 6, 45}, { 6, 50}, { 6, 55}, { 7,  0}, { 7,  5}, { 7, 10}, { 7, 15}, { 7, 20}, { 7, 25},
    { 7, 30}, { 7, 35}, { 7, 40}, { 7, 45}, { 7, 50}, { 7, 55}, { 8,  0}, { 8,  5}, { 8, 10}, { 8, 15},
    { 8, 20}, { 8, 25}, { 8, 30}, { 8, 35}, { 8, 40}, { 8, 45}, { 8, 50}, { 8, 55}, { 9,  0}, { 9,  5},
    { 9, 10}, { 9, 15}, { 9, 20}, { 9, 25}, { 9, 30}, { 9, 35}, { 9, 40}, { 9, 45}, { 9, 50}, { 9, 55},
    {10,  0}, {10,  5}, {10, 10}, {10, 15}, {10, 20}, {10, 25}, {10, 30}, {10, 35}, {10, 40}, {10, 45},
    {10, 50}, {10, 55}, {11,  0}, {11,  5}, {11, 10}, {11, 15}, {11, 20}, {11, 25}, {11, 30}, {11, 35}
  }
};

I would write code that stores or retrieves this in binary form in a file. It will be much faster than using JSON and requires less memory.

something like (typed here - not tested)

#include <LittleFS.h>
#include <Arduino.h>
#include <FS.h>  // For file operations

// Define the structs with packing
struct __attribute__((packed)) HourMinute {
  uint8_t hour;   // Using uint8_t for byte equivalent
  uint8_t minute; // Using uint8_t for byte equivalent
};

struct __attribute__((packed)) Configuration {
  bool flags[42];
  HourMinute timing[140];
};


Configuration config = {
  {
    true, true, false, false, true, true, false, false, true, true,
    false, false, true, true, false, false, true, true, false, false,
    true, true, false, false, true, true, false, false, true, true,
    false, false, true, true, false, false, true, true, false, false,
    true, true
  },
  {
    { 0,  0}, { 0,  5}, { 0, 10}, { 0, 15}, { 0, 20}, { 0, 25}, { 0, 30}, { 0, 35}, { 0, 40}, { 0, 45},
    { 0, 50}, { 0, 55}, { 1,  0}, { 1,  5}, { 1, 10}, { 1, 15}, { 1, 20}, { 1, 25}, { 1, 30}, { 1, 35},
    { 1, 40}, { 1, 45}, { 1, 50}, { 1, 55}, { 2,  0}, { 2,  5}, { 2, 10}, { 2, 15}, { 2, 20}, { 2, 25},
    { 2, 30}, { 2, 35}, { 2, 40}, { 2, 45}, { 2, 50}, { 2, 55}, { 3,  0}, { 3,  5}, { 3, 10}, { 3, 15},
    { 3, 20}, { 3, 25}, { 3, 30}, { 3, 35}, { 3, 40}, { 3, 45}, { 3, 50}, { 3, 55}, { 4,  0}, { 4,  5},
    { 4, 10}, { 4, 15}, { 4, 20}, { 4, 25}, { 4, 30}, { 4, 35}, { 4, 40}, { 4, 45}, { 4, 50}, { 4, 55},
    { 5,  0}, { 5,  5}, { 5, 10}, { 5, 15}, { 5, 20}, { 5, 25}, { 5, 30}, { 5, 35}, { 5, 40}, { 5, 45},
    { 5, 50}, { 5, 55}, { 6,  0}, { 6,  5}, { 6, 10}, { 6, 15}, { 6, 20}, { 6, 25}, { 6, 30}, { 6, 35},
    { 6, 40}, { 6, 45}, { 6, 50}, { 6, 55}, { 7,  0}, { 7,  5}, { 7, 10}, { 7, 15}, { 7, 20}, { 7, 25},
    { 7, 30}, { 7, 35}, { 7, 40}, { 7, 45}, { 7, 50}, { 7, 55}, { 8,  0}, { 8,  5}, { 8, 10}, { 8, 15},
    { 8, 20}, { 8, 25}, { 8, 30}, { 8, 35}, { 8, 40}, { 8, 45}, { 8, 50}, { 8, 55}, { 9,  0}, { 9,  5},
    { 9, 10}, { 9, 15}, { 9, 20}, { 9, 25}, { 9, 30}, { 9, 35}, { 9, 40}, { 9, 45}, { 9, 50}, { 9, 55},
    {10,  0}, {10,  5}, {10, 10}, {10, 15}, {10, 20}, {10, 25}, {10, 30}, {10, 35}, {10, 40}, {10, 45},
    {10, 50}, {10, 55}, {11,  0}, {11,  5}, {11, 10}, {11, 15}, {11, 20}, {11, 25}, {11, 30}, {11, 35}
  }
};


// File path to store the configuration
const char* configFilePath = "/config.bin";

// ---------------------------------------------------------
// ----------  Print the Configuration to Serial  ----------
// ---------------------------------------------------------
void printConfig(const Configuration& config) {

  Serial.println("Flags:");
  for (size_t i = 0; i < sizeof config.flags / sizeof * config.flags; ++i) {
    Serial.printf("\tFlag#%03zu\t%s\n", i + 1, config.flags[i] ? "true" : "false");
  }
  Serial.println();

  Serial.println("Timing");
  for (size_t i = 0; i < sizeof config.timing / sizeof * config.timing; ++i) {
    Serial.printf("\tTime#%03zu\t%02u:%02u\n", i + 1, config.timing[i].hour, config.timing[i].minute);
  }
}

// ---------------------------------------------------------
// ---------- Write the Configuration to LittleFS ----------
// ---------------------------------------------------------
void writeConfig(const Configuration& config) {
  File file = LittleFS.open(configFilePath, "wb");
  if (!file) {
    Serial.println("Failed to open file for writing");
    return;
  }

  size_t structSize = sizeof(Configuration);
  Serial.print("Writing ");
  Serial.print(structSize);
  Serial.println(" bytes to file");

  file.write(reinterpret_cast<const uint8_t*>(&config), sizeof(Configuration));
  file.close();

  Serial.println("Configuration written to file");
}

// ---------------------------------------------------------
// ---------- Read the Configuration from LittleFS ----------
// ---------------------------------------------------------
void readConfig(Configuration& config) {
  File file = LittleFS.open(configFilePath, "rb");
  if (!file) {
    Serial.println("Failed to open file for reading");
    return;
  }

  // Check file size
  size_t fileSize = file.size();
  size_t structSize = sizeof(Configuration);

  if (fileSize != structSize) {
    Serial.print("File size mismatch: expected ");
    Serial.print(structSize);
    Serial.print(" bytes, but got ");
    Serial.println(fileSize);
    file.close();
    return;
  }

  Serial.print("Reading ");
  Serial.print(structSize);
  Serial.println(" bytes from file");

  file.read(reinterpret_cast<uint8_t*>(&config), structSize);
  file.close();

}


void setup() {
  Serial.begin(115200);
  if (!LittleFS.begin(true)) { // Pass true to format the file system
    Serial.println("Failed to format LittleFS");
    return;
  }
  Serial.println("LittleFS formatted and mounted");

  delay(2000); // later in code...

  // Example of writing and reading config
  writeConfig(config);

  delay(2000); // later in code...

  Configuration configFromFile;

  readConfig(configFromFile);
  Serial.println("Configuration read from file");
  printConfig(configFromFile);
}

void loop() {}

and if you want to access specific indexes in the arrays with a name, juste have some #define or an enum with the indexes and then you can use the names instead of a hardcoded value.

davidmarli · August 17, 2024, 9:49am

Ok, so I explain.
I work on an ESP32 (ttgo, only 4mB)
On this I have html files in littleFS partition and my program.

On an html page, the user can schedule 2 Switches (you have to say when you want it ON)

(RELAIS 1, is for switch 1) html page contains the same thing for Switch 2.

When the user click on "Apply" button the form is sent to esp32.

And that's why is use json file : when I get the values from html, I know how to do it whith json files.
But perhaps is there an another way to save the data from html page directly on littleFS partition file

noiasca · August 17, 2024, 9:53am

do you get a JSON from the browser when the user presses Apply or the parameters as "standard" POST/GET parameters?

J-M-L · August 17, 2024, 10:01am

The JSON sent from the browser would need to be minimised to reduce trafic an memory — you don't need labels — it could mimic the struct

{
  "bools": [true, true, false, false, true, true, false, false, true, true, false, false, true, true, false, false, true, true, false, false, true, true, false, false, true, true, false, false, true, true, false, false, true, true, false, false],
  "times": [
    [0, 0], [0, 5], [0, 10], [0, 15], [0, 20], [0, 25], [0, 30], [0, 35], [0, 40], [0, 45],
    [0, 50], [0, 55], [1, 0], [1, 5], [1, 10], [1, 15], [1, 20], [1, 25], [1, 30], [1, 35],
    [1, 40], [1, 45], [1, 50], [1, 55], [2, 0], [2, 5], [2, 10], [2, 15], [2, 20], [2, 25],
    [2, 30], [2, 35], [2, 40], [2, 45], [2, 50], [2, 55], [3, 0], [3, 5], [3, 10], [3, 15],
    [3, 20], [3, 25], [3, 30], [3, 35], [3, 40], [3, 45], [3, 50], [3, 55], [4, 0], [4, 5],
    [4, 10], [4, 15], [4, 20], [4, 25], [4, 30], [4, 35], [4, 40], [4, 45], [4, 50], [4, 55],
    [5, 0], [5, 5], [5, 10], [5, 15], [5, 20], [5, 25], [5, 30], [5, 35], [5, 40], [5, 45],
    [5, 50], [5, 55], [6, 0], [6, 5], [6, 10], [6, 15], [6, 20], [6, 25], [6, 30], [6, 35],
    [6, 40], [6, 45], [6, 50], [6, 55], [7, 0], [7, 5], [7, 10], [7, 15], [7, 20], [7, 25],
    [7, 30], [7, 35], [7, 40], [7, 45], [7, 50], [7, 55], [8, 0], [8, 5], [8, 10], [8, 15],
    [8, 20], [8, 25], [8, 30], [8, 35], [8, 40], [8, 45], [8, 50], [8, 55], [9, 0], [9, 5],
    [9, 10], [9, 15], [9, 20], [9, 25], [9, 30], [9, 35], [9, 40], [9, 45], [9, 50], [9, 55],
    [10, 0], [10, 5], [10, 10], [10, 15], [10, 20], [10, 25], [10, 30], [10, 35], [10, 40], [10, 45],
    [10, 50], [10, 55], [11, 0], [11, 5], [11, 10], [11, 15], [11, 20], [11, 25], [11, 30], [11, 35]
  ]
}

and you would parse the incoming JSON and save in the struct / file system.

davidmarli · August 17, 2024, 10:06am

When the user click on apply button, I use this :

 <input type="submit" value=" Appliquer les parametres" class="btn btn-primary btn-user border-dark btn-block col-4 mx-auto shadow-lg h-100" style = "font-size:20px">

That's just a submit, and send a POST

noiasca · August 17, 2024, 10:30am

does this mean you read several single parameters with .hasParameter() and/or .arg()? so you DON'T receive a JSON from the browser?

rsmls · August 17, 2024, 11:06am

Read the file contents, parse into variables as you're doing now. Same approach, just not using json libraries. It'll be more memory-efficient to boot, if you only use/make the functionality you need.

Yup, that would be one way of doing it.

Or an integer representing minutes from midnight. It'll fit into 2 bytes, making it even smaller than a typical HHMM format.

That would work, too.

Or use a bitfield for the bools. Bit-fields - cppreference.com
Very memory-efficient.

No json required for that. You get POST or GET data (whichever you prefer; boils down to pretty much the same from a parsing viewpoint) that you can parse in whatever way you want. Json just adds a layer of complication you don't need.

You could also compact all data into a single string/character array on the browser side (using javascript) and then unpack it server-side to parse the variables you need. It's not terribly complicated JavaScript that you need for this and it's basically a variant of C anyway, so you're already familiar with the basics.

I have a lighting system running here that works this way. There's a simple web form that can be used to modify the configuration (4 programs for 3 light sources with 4 channels each, so 64 data elements in total, plus an array of booleans which channels are presently on). On the browser side, all fields are combined together and packed into a single character array that's passed as a GET variable, then the ESP server unpacks the string and parses it into the separate veriables. It's all done iteratively so the number of lines of code is limited and it doesn't involve the highly repetitive/redundant structure you've shown in your snippet.

davidmarli · August 17, 2024, 11:13am

/*****************************************************************/
/***** Application des Paramètres relais via INTERFACE WEB   *****/
/*****************************************************************/
void page_SAVEconfigRELAIS(AsyncWebServerRequest *request)
{

configrelais.NO=false;

for(int i=0;i<42;i++) {configrelais.CheckJoursTEMPO[i] = false;}
for(int i=0;i<70;i++) {configrelais.StartHour[i] = "99:99";configrelais.EndHour[i] = "99:99"; }
Serial.println(F("Page config RELAIS"));
int params = request->params();
for(int i=0;i<params;i++){
    AsyncWebParameter* p = request->getParam(i);
    Serial.printf("POST[%s]: %s\n", p->name().c_str(), p->value().c_str());
    
    if (strstr(p->name().c_str(), "NO")) {configrelais.NO=true;} 

    if (strstr(p->name().c_str(), "ip_adress")) {configrelais.ip_adresse_delesteur.fromString(p->value().c_str()); }

    if (strstr(p->name().c_str(), "MaxTimeManu")) configrelais.MaxTimeManu=p->value().c_str();

// COULEUR JOURS RELAIS 1
if (strstr(p->name().c_str(), "CJR_Lun1")) configrelais.CheckJoursTEMPO[0]=true;
if (strstr(p->name().c_str(), "CJW_Lun1")) configrelais.CheckJoursTEMPO[1]=true;
if (strstr(p->name().c_str(), "CJB_Lun1")) configrelais.CheckJoursTEMPO[2]=true;

if (strstr(p->name().c_str(), "CJR_Mar1")) configrelais.CheckJoursTEMPO[3]=true;
if (strstr(p->name().c_str(), "CJW_Mar1")) configrelais.CheckJoursTEMPO[4]=true;
if (strstr(p->name().c_str(), "CJB_Mar1")) configrelais.CheckJoursTEMPO[5]=true;

if (strstr(p->name().c_str(), "CJR_Mer1")) configrelais.CheckJoursTEMPO[6]=true;
if (strstr(p->name().c_str(), "CJW_Mer1")) configrelais.CheckJoursTEMPO[7]=true;
if (strstr(p->name().c_str(), "CJB_Mer1")) configrelais.CheckJoursTEMPO[8]=true;

if (strstr(p->name().c_str(), "CJR_Jeu1")) configrelais.CheckJoursTEMPO[9]=true;
if (strstr(p->name().c_str(), "CJW_Jeu1")) configrelais.CheckJoursTEMPO[10]=true;
if (strstr(p->name().c_str(), "CJB_Lun1")) configrelais.CheckJoursTEMPO[11]=true;

if (strstr(p->name().c_str(), "CJR_Ven1")) configrelais.CheckJoursTEMPO[12]=true;
if (strstr(p->name().c_str(), "CJW_Ven1")) configrelais.CheckJoursTEMPO[13]=true;
if (strstr(p->name().c_str(), "CJB_Ven1")) configrelais.CheckJoursTEMPO[14]=true;

if (strstr(p->name().c_str(), "CJR_Sam1")) configrelais.CheckJoursTEMPO[15]=true;
if (strstr(p->name().c_str(), "CJW_Sam1")) configrelais.CheckJoursTEMPO[16]=true;
if (strstr(p->name().c_str(), "CJB_Sam1")) configrelais.CheckJoursTEMPO[7]=true;

noiasca · August 17, 2024, 11:14am

HHMM 2359 fits as good in a two byte integer as 1140 for 60*24 minutes.

Topic		Replies	Views
Webapp mit JSON und SPIFFS -> eure Expertise Deutsch	76	1926	July 15, 2023
Need help with json object Programming	12	3343	December 4, 2021
Sauvgarder JSON dans le SPIFFS d'un nodemcu Français	24	1521	November 23, 2021
Reading JSON file and array General Guidance	27	2580	July 25, 2023
Parse Json download size Programming	54	2904	July 24, 2022

How to minimize the size of a json file

Related topics