Fill an array with string from SPIFFS file

Projects Programming
1

Hi guys! I've got a file.txt in my SPIFFS where I store 5 row of json object in this format

{"AcX": 1625, "AcY": -225, "AcZ": 1235, "time": 3000}
{"AcX": 1625, "AcY": -25, "AcZ": 1556, "time": 6000}
{"AcX": 1625, "AcY": -132, "AcZ": 1789, "time": 9000}
{"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}
{"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}

After I filled my file.txt with my 5 rows I call a function that read the file, store every row in the array and then I send it via mqtt to my topic.

void mqtt_publish(){
  //Open and read the file
  File file = LittleFS.open("/file.txt", "r");
  if(!file){
    Serial.println("file open failed");
  } //Serial.println("------ Reading ------");
  //Create an empty array
  JsonArray arr = doc.to<JsonArray>();
  for (int i=0; i<=5; i++){
    String s = file.readStringUntil('\n');
    JsonObject obj = doc.createNestedObject();
    doc[i].add(serialized(s));
  }
  char sendread[MQTT_BUFFER];
  serializeJsonPretty(doc, sendread);
  int ret = client.publish("esp8266/JSON", sendread);
  
  file.close();
}

Anyway the output in the serial monitor is this

12:47:54.208 -> Message arrived in topic: esp8266/JSON
12:47:54.208 -> Message:[
12:47:54.208 ->   {},
12:47:54.208 ->   {},
12:47:54.208 ->   {},
12:47:54.208 ->   {},
12:47:54.208 ->   {},
12:47:54.208 ->   {}
12:47:54.208 -> ]

The array with the 5 elements it's created but not filled. What am I missing? Thanks a lot!

2

You have to show the code where you prepare the JSON object for printing to the serial console.

3

That's the callback

void callback(char* topic, byte* payload, unsigned int length) {

  Serial.print("Message arrived in topic: ");
  Serial.println(topic);
  Serial.print("Message:");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
  Serial.println("-----------------------"); 
}
4

what's doc?  JsonArray arr = doc.to<JsonArray>();what's the purpose of arr?

Don't post snippets please (Snippets R Us!)

5

Not best practice to print from the callback but...shrug.

void IRAM_ATTR mqttCallback(char* topic, byte * payload, unsigned int length)
{
  // clear locations
  memset( x_message.payload, '\0', 150 );
  x_message.topic = ""; //clear string buffer
  x_message.topic = topic;
  int i = 0;
  for ( i; i < length; i++)
  {
    x_message.payload[i] = ((char)payload[i]);
  }
  x_message.payload[i] = '\0';
  //log_i( "topic %s payload %s" ,x_message.topic, x_message.payload );
  xQueueOverwrite( xQ_Message, (void *) &x_message );// send data
} // void mqttCallback(char* topic, byte* payload, unsigned int length)

I uncomment the //log_i( "topic %s payload %s" ,x_message.topic, x_message.payload ); line to print. Remember printing is a time consuming effort by the MCU.

Spoted the thing your missing in your payload parsing?

I noticed you do the thing a bit differently.

I create the message structure here:

struct stu_message
{
  char payload [150] = {'\0'};
  String topic;
} x_message;

In setup I limit the string size to make a buffer and prevent string memory fragmentations.

void setup()
{
x_message.topic.reserve(150);
}
6

Yep I know @Idahowalker, infact I only need to send the array via mqtt, and in my function I wrote

char sendread[MQTT_BUFFER];
serializeJsonPretty(doc, sendread);
int ret = client.publish("esp8266/JSON", sendread);

I was printing on the serial monitor just to check the result
7

Idahowalker:
Not best practice to print from the callback but...shrug.

void IRAM_ATTR mqttCallback(char* topic, byte * payload, unsigned int length)

{
 // clear locations
 memset( x_message.payload, '\0', 150 );
 x_message.topic = ""; //clear string buffer
 x_message.topic = topic;
 int i = 0;
 for ( i; i < length; i++)
 {
   x_message.payload[i] = ((char)payload[i]);
 }
 x_message.payload[i] = '\0';
 //log_i( "topic %s payload %s" ,x_message.topic, x_message.payload );
 xQueueOverwrite( xQ_Message, (void ) &x_message );// send data
} // void mqttCallback(char topic, byte* payload, unsigned int length)

Wow, that's a lot. I'm a newbie to arduino an C programming. It's quite different the syntax I'm not sure what I'm looking at. There's nothing easier for me to solve my problem? I still don't get what am I missing

8

What I understand is that the message has been sent but when the receiver callback gets triggered a receipt of data is not indicated.

Find, download, and install MQTT.fx. Connect to your broker and see what is being published by going to the subscribe thingy, scan for posted topics. See the topic in the scan, click on it to see the payload. Id it empty, sending issue. If their is data then receiving issue.

Prove sender or receiver as being as fault.

9

Idahowalker:
What I understand is that the message has been sent but when the receiver callback gets triggered a receipt of data is not indicated.

Find, download, and install MQTT.fx. Connect to your broker and see what is being published by going to the subscribe thingy, scan for posted topics. See the topic in the scan, click on it to see the payload. It it empty, sending issue. If their is data then receiving issue.

Prove sender or receiver as being as fault.

Ok that's the problem, even in the topic the message is empty. So the problem it's in the sending function, but I can't see what's causing it

[color=#b72828][
 {},
 {},
 {},
 {},
 {},
 {}
][/color]
10

You create arr and obj but never use them, I think you should look at the ArduinoJson documentation

11

guix:
You create arr and obj but never use them, I think you should look at the ArduinoJson documentation

JsonArray | ArduinoJson 6
JsonArray::createNestedObject() | ArduinoJson 6

StaticJsonDocument<200> doc;
JsonArray array = doc.to();
JsonObject nested = array.createNestedObject();
nested["hello"] = "world";
serializeJson(array, Serial);

With the difference I was trying to fill the array through a for loop

That's how I declared the capacity and the array

//compute the required size
const size_t CAPACITY = JSON_ARRAY_SIZE(5) + 5*JSON_OBJECT_SIZE(4);
//allocate the memory for the document
StaticJsonDocument<CAPACITY> doc;
12

With the difference I was trying to fill the array through a for loop

as asked in #3, you don't use the array - so you are not filling it through a for loop...where do you see arr in

  for (int i=0; i<=5; i++){
    String s = file.readStringUntil('\n');
    JsonObject obj = doc.createNestedObject();
    doc[i].add(serialized(s));
  }
13

J-M-L:
as asked in #3, you don't use the array - so you are not filling it through a for loop...where do you see arr in

  for (int i=0; i<=5; i++){

String s = file.readStringUntil('\n');
   JsonObject obj = doc.createNestedObject();
   doc[i].add(serialized(s));
 }

Ok I understand, but how do I fill the array? I'm getting confused by all the nested staff

14

if you really need to have separate lines, each being a little JSON object, then I would just have a global doc that will be the end result and add in each line.

==> read a line, build a temporary JSON by de-serializing that line, and add it to the global doc.
Once you've finish reading the file then the global doc holds a JSON array with all your lines.

something like this. Typed here, not tested... give it a try

#include <SPIFFS.h>
#include <ArduinoJson.h>

DynamicJsonDocument globalDoc(1000);

void setup() {
  Serial.begin(115200);

  if (!SPIFFS.begin()) {
    Serial.println("An Error has occurred while mounting SPIFFS");
    while (true) yield();
  }

  File file = SPIFFS.open("/file.txt", "r");
  if (!file) {
    Serial.println("Failed to open test file");
    while (true) yield();
  } else {
    String jsonLine;
    jsonLine.reserve(100);
    StaticJsonDocument<100> localDoc;
    while (file.available()) {
      jsonLine = file.readStringUntil('\n');
      DeserializationError error = deserializeJson(localDoc, jsonLine);
      if (!error) globalDoc.add(localDoc);
    }
    serializeJsonPretty(globalDoc, Serial);
    file.close();
  }
}

void loop() {}

if all works OK, Serial Monitor (@ 115200 bauds) will show something like this (give or take, you might have each entry on a separate line)

[color=purple]
[b][color=red][[/color][/b]
  {"AcX": 1625, "AcY": -225, "AcZ": 1235, "time": 3000}[b][color=red],[/color][/b]
  {"AcX": 1625, "AcY": -25, "AcZ": 1556, "time": 6000}[b][color=red],[/color][/b]
  {"AcX": 1625, "AcY": -132, "AcZ": 1789, "time": 9000}[b][color=red],[/color][/b]
  {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}[b][color=red],[/color][/b]
  {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}
[b][color=red]][/color][/b]
[/color]

if you are in charge of the format of the SPIFF file, then why don't you make it a true JSON array in the first place and make it look like the above. This way you could just create your JSON donc in one godeserializeJson(doc, file);

PS: of course the size of the global and local JSON docs need to be adapted to the requirements.

15

J-M-L:
if you really need to have separate lines, each being a little JSON object, then I would just have a global doc that will be the end result and add in each line.

==> read a line, build a temporary JSON by de-serializing that line, and add it to the global doc.
Once you've finish reading the file then the global doc holds a JSON array with all your lines.

something like this. Typed here, not tested... give it a try

#include <SPIFFS.h>

#include <ArduinoJson.h>

DynamicJsonDocument globalDoc(1000);

void setup() {
 Serial.begin(115200);

if (!SPIFFS.begin()) {
   Serial.println("An Error has occurred while mounting SPIFFS");
   while (true) yield();
 }

File file = SPIFFS.open("/file.txt", "r");
 if (!file) {
   Serial.println("Failed to open test file");
   while (true) yield();
 } else {
   String jsonLine;
   jsonLine.reserve(100);
   StaticJsonDocument<100> localDoc;
   while (file.available()) {
     jsonLine = file.readStringUntil('\n');
     DeserializationError error = deserializeJson(localDoc, jsonLine);
     if (!error) globalDoc.add(localDoc);
   }
   serializeJsonPretty(globalDoc, Serial);
   file.close();
 }
}

void loop() {}



if all works OK, Serial Monitor (@ 115200 bauds) will show something like this (give or take, you might have each entry on a separate line)


[
 {"AcX": 1625, "AcY": -225, "AcZ": 1235, "time": 3000},
 {"AcX": 1625, "AcY": -25, "AcZ": 1556, "time": 6000},
 {"AcX": 1625, "AcY": -132, "AcZ": 1789, "time": 9000},
 {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000},
 {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}
]




if you are in charge of the format of the SPIFF file, then why don't you make it a true JSON array in the first place and make it look like the above. This way you could just create your JSON donc in one go

deserializeJson(doc, file);




PS: of course the size of the global and local JSON docs need to be adapted to the requirements.

Ok it seems to work, not fully but I hope I'll manage to solve other issues.

Just one last question! I am in charge of the format of the SPIFFS, but this is the first time with SPIFFS and also arduino programming so I didn't know how to manage all these things. So how could I create my doc as a JSON doc in one go?

16

edward_radical:
how could I create my doc as a JSON doc in one go?

Not sure I understand the question.

JSON is a documented format.

JSON defines only two data structures: objects and arrays.
An object is a set of name-value pairs, and an array is a list of values (amongst seven value types: string, number, object, array, true, false, and null)

So decide how you wan to represent the JSON data

For example this would be one array (using []) holding multiple objects (using {}), each object having 4 properties (name-value pairs): AcX, AcY, AcZ and time

[
  {"AcX": 1625, "AcY": -225, "AcZ": 1235, "time": 3000},
  {"AcX": 1625, "AcY": -25, "AcZ": 1556, "time": 6000},
  {"AcX": 1625, "AcY": -132, "AcZ": 1789, "time": 9000},
  {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000},
  {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}
]

You could make it more compact as the labels are useless (you can access through indexes)

[
  [1625, -225, 1235, 3000],
  [1625, -25, 1556, 6000],
  [1625, -132, 1789, 9000],
  [1625, -67, 345, 12000],
  [1625, -67, 345, 12000]
]

so this would be an array of array holding numbers

17

J-M-L:
if you are in charge of the format of the SPIFF file, then why don't you make it a true JSON array in the first place and make it look like the above. This way you could just create your JSON donc in one go

deserializeJson(doc, file);

PS: of course the size of the global and local JSON docs need to be adapted to the requirements.

I was referring at this

18

well if the file is a JSON format already then it's really just that line... So the original code would become

#include <SPIFFS.h>
#include <ArduinoJson.h>

DynamicJsonDocument globalDoc(1000);

void setup() {
  Serial.begin(115200);

  if (!SPIFFS.begin()) {
    Serial.println(F("An Error has occurred while mounting SPIFFS"));
    while (true) yield();
  }

  File file = SPIFFS.open("/file.txt", "r");
  if (!file) {
    Serial.println(F("Failed to open test file"));
    while (true) yield();
  } else {
    DeserializationError error deserializeJson(globalDoc, file); // <=== READ THE FILE AS A JSON 
    file.close();
    if (!error) serializeJsonPretty(globalDoc, Serial); 
    else Serial.println(F("Failed to read JSON"));
  }
}

void loop() {}
19

J-M-L:
if you really need to have separate lines, each being a little JSON object, then I would just have a global doc that will be the end result and add in each line.

==> read a line, build a temporary JSON by de-serializing that line, and add it to the global doc.
Once you've finish reading the file then the global doc holds a JSON array with all your lines.

something like this. Typed here, not tested... give it a try

#include <SPIFFS.h>

#include <ArduinoJson.h>

DynamicJsonDocument globalDoc(1000);

void setup() {
 Serial.begin(115200);

if (!SPIFFS.begin()) {
   Serial.println("An Error has occurred while mounting SPIFFS");
   while (true) yield();
 }

File file = SPIFFS.open("/file.txt", "r");
 if (!file) {
   Serial.println("Failed to open test file");
   while (true) yield();
 } else {
   String jsonLine;
   jsonLine.reserve(100);
   StaticJsonDocument<100> localDoc;
   while (file.available()) {
     jsonLine = file.readStringUntil('\n');
     DeserializationError error = deserializeJson(localDoc, jsonLine);
     if (!error) globalDoc.add(localDoc);
   }
   serializeJsonPretty(globalDoc, Serial);
   file.close();
 }
}

void loop() {}



if all works OK, Serial Monitor (@ 115200 bauds) will show something like this (give or take, you might have each entry on a separate line)


[
 {"AcX": 1625, "AcY": -225, "AcZ": 1235, "time": 3000},
 {"AcX": 1625, "AcY": -25, "AcZ": 1556, "time": 6000},
 {"AcX": 1625, "AcY": -132, "AcZ": 1789, "time": 9000},
 {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000},
 {"AcX": 1625, "AcY": -67, "AcZ": 345, "time": 12000}
]




if you are in charge of the format of the SPIFF file, then why don't you make it a true JSON array in the first place and make it look like the above. This way you could just create your JSON donc in one go

deserializeJson(doc, file);




PS: of course the size of the global and local JSON docs need to be adapted to the requirements.

Everything works this way, I have just one last question. What should I do if I want to send the content of globalDoc to my topic via mqtt? I tried this but doesn't work:

void mqtt_publish(){
  File file = LittleFS.open("/file.txt", "r");
  if(!file){
    Serial.println("file open failed");
    while (true) yield();
  }else {
    DynamicJsonDocument globalDoc(1000);
    String jsonLine;
    jsonLine.reserve(100);
    StaticJsonDocument <100> localDoc;
    while (file.available()) {
      jsonLine = file.readStringUntil('\n');
      DeserializationError error = deserializeJson(localDoc, jsonLine);
      if (!error) globalDoc.add(localDoc);  
    }
    
    char sendread[MQTT_BUFFER];
    serializeJsonPretty(globalDoc, sendread);
    int ret = client.publish("esp8266/JSON", sendread);
    file.close();
  }
}

I added the last 4 lines after what you suggested

20

please don't quote full messages. This is totally useless and makes reading the thread cumbersome. My text is already readable above

To your question, have you looked at the extensive documentation of the Arduino JSON library? try this

char buffer[1024]; // adjust to what's needed (should be the file size)
size_t n = serializeJson(globalDoc, buffer);
client.publish("esp8266/JSON", buffer, n);

but if this is the only thing you do with globalDoc, as your file is already in JSON format now, there is no need to deserialize and serialize... just read the file into the char buffer and publish it. You'll save all the memory needed by JSON stuff