Hello everyone,
I have a json file stored in an sd card. The problem is that this json file is very badly composed. I try to recompose it to finally have a json file that looks like this:
(this is an excerpt, there will be many more points and hours)
The json file doesn't have the correct syntax, but that's for example.
I manage to create a json file which contains the point objects (coordinates) with a random altitude. But then I can't add the real altitudes, nor the hours...
Basically, once the file is created as in the following code, it cannot be modified.
#include <SPI.h>
#include <SD.h>
#include <ArduinoJson.h>
File dataFile;
int i=1;
byte nx;
byte ny;
int start;
int end;
void setup() {
Serial.begin(9600);
while (!Serial) {}
}
void loop() {
if(i==1){
if (!SD.begin(SS)) {
Serial.println("SD card initialization error.");
return;
}
dataFile = SD.open("/altjson.txt", FILE_READ);
if (!dataFile) {
Serial.println("Error opening file.");
return;
}
String unit = "";
float lo1=0.0;
float la1=0.0;
while (dataFile. available()) {
String line = dataFile.readStringUntil('\n');
int index = line.indexOf("parameterUnit\":");
if (index != -1) {
start = index + 16; // unit start index
end = line.indexOf('"', start); // end index of unit
unit = line.substring(start, end);
}
int index2 = line.indexOf("lo1\":");
if (index2 != -1) {
start = index2 + 5;
end = line.indexOf(',', start);
lo1 = line.substring(start, end).toFloat();
}
int index3 = line.indexOf("la1\":");
if (index3 != -1) {
start = index3 + 5;
end = line.indexOf(',', start);
la1 = line.substring(start, end).toFloat();
}
int index4 = line.indexOf("nx\":");
if (index4 != -1) {
start = index4 + 4;
end = line.indexOf(',', start);
nx = line.substring(start, end).toFloat();
}
int index5 = line.indexOf("ny\":");
if (index5 != -1) {
start = index5 + 4;
end = line.indexOf(',', start);
ny = line.substring(start, end).toFloat();
}
}
Serial.println("Parameter unit: " + unit);
Serial.println("Value of lo1: " + String(lo1));
Serial.println("Value of la1: " + String(la1));
Serial.println("Value of nx: " + String(nx));
Serial.println("Value of ny: " + String(ny));
DynamicJsonDocument doc(4096);
JsonArray datameteo = doc.createNestedArray("datameteo");
float la = la1;
// Add objects for each value of:
for(int j=0; j < ny; j++){
float lo = lo1;
// Add objects for each value of lo:
for (int i = 0; i < nx; i++) {
JsonObject obj = datameteo.createNestedObject();
Serial.println(lo);
String str = String(la) + ";" + String(lo);
JsonArray altitudes = obj.createNestedArray(str);
JsonObject altitudeObj = altitudes.createNestedObject();
altitudeObj["altitude"] = random(100, 200);
lo+=0.01;
}
la-=0.01;
}
// Open the newjson.txt file in write mode
File file = SD.open("/newjson.txt", FILE_WRITE);
if (!file) {
Serial.println("Failed to open file.");
return;
}
// Write the JSON array to the file
serializeJson(doc, file);
// Close the file
file.close();
Serial.println("File created successfully!");
delay(2000);
}
}
How to modify the json created( JsonArray datameteo) to add altitude objects, other arrays in geo-point arrays, modify the values? If outputting to JSON is too complicated, arrays[] are fine as well.
I modified the json as I want it to be, with a valid construct this time.
Of course, if storing in an array is simpler, a similar structure would be fine as well.
If you want to store the data in a file, JSON .txt is convenient and the structure is very simple.
Please explain what you mean by "storing in an array". For program code data with a variety of variable types, structures are very useful. But for readable text file storage, structures generally require custom code to convert them into ASCII.
I'm not necessarily looking to have something human readable. But something whose data can be retrieved via a quick and suitable path by a machine.
Ex: coordinates -> time -> temperature value
coordinates -> return altitude
The original file is quite poorly made which makes the data difficult to obtain optimally.
But the problem is that I can't transform the base file into a json as I want.
I say that the dataset can also be in the form of arrays[], because anyway the file will be read by a microcontroller that I would program myself
An array of structures would be required. You could write them out or read them back in as binary data to an SD card.
If you can read the base file with no problem, then declare a suitable structure for one record. The structure will not have the flexibility of a JSON object, though.
The code that created the base file you posted needs to be fixed, but there are only a couple of minor errors. I fixed the obvious ones. The following is valid and can be decoded using this: JSON Decode Online to decode JSON to readable form.
I didn't quite understand what the JSON editor tool could do for me.
My problem and that I can't get a recomposed JSON like I showed.
The best I can do is:
I managed to get a json like this (with the code from post #1) but then I can't change the altitudes, add new json objects in those of the coordinate points...
I do not know how to do. In the for loop, it pretty much works. Outside, unable to access or modify the json file.
Why do you want to recompose on an Arduino, with such limited memory and so few debugging tools?
Seems like this is a job for a PC data base editor, e.g. SQL. Read all the data in, and write back out only those parts you want, reindexed/reordered and formatted the way you like.
What program produced that JSON data file? Surely there is some flexibility there.
In fact this work could be done by a computer. However, I don't want it to complicate. Processing by a microcontroller seemed easier to me.
I'm not exactly using an arduino but an esp32 s2 which has much better computing capabilities.
But if processing a json file on a regular server/computer is easier, then I'll have to get on with it.
If all goes well I should create a server that does this:
Asks the user for a geographical area and a time range -> Downloads meteorological data files (GRIB2) -> Selects and combines useful data from these files -> Converts GRIB2 to JSON
So we could add:
-> Convert bad JSON to optimal JSON
If you recommend this method, what tools should I use to convert my json as I want? Python, SQL, which is the simplest and most suitable in your opinion?
You decide. It is clear that you are having trouble writing a C/C++ program for ESP32 to do what you want.
If you want to hire help writing code, post on the Jobs and Paid Collaboration forum section. Seems like it might be an interesting challenge to someone.