How can i use ArdoinoJSON with an own wrapper

djpx · June 10, 2022, 11:39am

I would like to write alittle wrapper around the ArdoinoJSON libery with the stuff which I need.
The Problem is that I got some issues with the StaticJsonDocument since it needs an const size. E.g. StaticJsonDocument<800> jsonDocument;
But i can not take a variable like
int size = string.length()
only
int size = 100

So I wrote this testfile for me:



#include <ArduinoJson.h>
#include "Utils/Map.h"
#include "Utils/MapEntry.h"

void setup()
{
  StaticJsonDocument<200> doc;

  doc["sensor"] = "gps";
  doc["time"] = 1351824120;
  doc["data"][0] = 48.756080;
  doc["data"][1] = 2.302038;

  serializeJson(doc, Serial);

  String temp = jsonToString(doc);
  stringToJson(temp);
}

JsonDocument *stringToJson(String jsonString, bool useDynamicDocument = false)
{
  const int documentSize = jsonString.length();
  if (useDynamicDocument)
  {
    const int documentSize = jsonString.length();
    DynamicJsonDocument jsonDocument(documentSize);
    deserializeJson(jsonDocument, jsonString);
    return &jsonDocument;
  }
  else
  {
    if (documentSize <= 50)
    {
      StaticJsonDocument<50> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 100)
    {
      StaticJsonDocument<100> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 200)
    {
      StaticJsonDocument<200> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 300)
    {
      StaticJsonDocument<300> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 400)
    {
      StaticJsonDocument<400> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 400)
    {
      StaticJsonDocument<400> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 400)
    {
      StaticJsonDocument<400> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 500)
    {
      StaticJsonDocument<500> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 600)
    {
      StaticJsonDocument<600> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 700)
    {
      StaticJsonDocument<700> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 800)
    {
      StaticJsonDocument<800> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 900)
    {
      StaticJsonDocument<900> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    else if (documentSize <= 1024)
    {
      StaticJsonDocument<1024> jsonDocument;
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
    // If stringsize is too big for the StaticJsonDocument
    else
    {
      DynamicJsonDocument jsonDocument(documentSize);
      deserializeJson(jsonDocument, jsonString);
      return &jsonDocument;
    }
  }
}



String jsonToString(DynamicJsonDocument jsonDocument)
{
  char *jsonString;
  serializeJson(jsonDocument, jsonString);
  return jsonString;
}

// Does not work because it would like to have the <number> on the StaticJsonDocument
//error: missing template arguments before 'jsonDocument'
// It is possible to change it to JsonDocument but then I can not call it anymore

String jsonToString(StaticJsonDocument jsonDocument)
{
  char *jsonString;
  serializeJson(jsonDocument, jsonString);
  return jsonString;
}

void loop()
{
}

Is there a way to make this code work somehow?

J-M-L · June 10, 2022, 11:44am

The easiest way to deal with it if your code needs to accept up to 1024 is to allocate the 1024 bytes and be done with it.

There is little point saving that memory if the code is supposed to work when you need it.... if you get short in SRAM with 1024 with the rest of the program running, then it means your code can't support that anyway.

Also objects allocated in a block get freed once the block execution is completed

djpx · June 10, 2022, 12:11pm

My code is more used as an little helper so it automaticly decides which memory allocation it should use (stack or heap) since I never really know how big the data is. It can range from 50 to 10k or more. I could do it really easy and just use the

DynamicJsonDocument

Because it could store smaller jsonsdocuments too.
" DynamicJsonDocument allows storing much larger documents than StaticJsonDocument because it is not limited by the size of the stack."

But it is recommended to use the static version for smaller documents.
"You can choose to store your JsonDocument in the stack or in the heap:

Use a StaticJsonDocument to store in the stack (recommended for documents smaller than 1KB)
Use a DynamicJsonDocument to store in the heap (recommended for documents larger than 1KB) "

And the static document can be allocated faster with less overhead and less memory fragmentation thats why I like to use it for smaller documents.

If I always would allocate 1024 my ram would have fast ram fragmentation and I would just loose alot of resources/time. (And i could not process the big json documents)

Danois90 · June 10, 2022, 12:36pm

Which hardware are you using? If memory fragmentation is a concern, then you should only allocate one global instance of JsonDocument with all the memory you require and then re-use that instance throughout your code:

StaticJsonDocument<10240> jsonDoc;

void setup()
{
  jsonDoc.clear();
  jsonDoc["sensor"] = "gps";
  jsonDoc["time"] = 1351824120;
  jsonDoc["data"][0] = 48.756080;
  jsonDoc["data"][1] = 2.302038;

  serializeJson(jsonDoc, Serial);

  String temp = jsonToString(jsonDoc);
  stringToJson(temp);
}

...

J-M-L · June 10, 2022, 12:43pm

it would be statically allocated so it happens once and could be done in the setup but I now understand that you want to be able to deal with any size and pick a static document when it's small and a dynamic one when it's large.

the challenge you have is that what you allocate in the function is not persistant. The object is freed up when the object goes out of scope.

try this code for example (➜ in the wokwi simulator)

class Buffer {
  public:
    Buffer(size_t s) {
      Serial.println(F("constructor"));
      if (s < 100) size = 100;
      else if (s < 200) size = 200;
      else size = 300;
    }

    ~Buffer() {
      Serial.print(F("destructor for size: "));
      Serial.println(size);
    }

  private:
    size_t size;
};

Buffer* createBuffer(size_t n) {
  Serial.println(F("entering function"));
  Buffer b(n);
  Serial.println(F("just before return"));
  return &b;
}

void setup() {
  Serial.begin(115200); Serial.println();
  Serial.println(F("before createBuffer"));
  createBuffer(80);
  Serial.println(F("after createBuffer"));
}

void loop() {}

you'll see that even though the compiler lets me return the local object the destructor has been called and the pointer you return is not good to use.

if you compile with all warnings on, you should get an alert

warning: address of local variable 'b' returned [-Wreturn-local-addr]

that should be your clue it's not good.

djpx · June 11, 2022, 6:09pm

I am using an ESP32S2.
According to the webside it is not recommended to reuse the document.

Since I need it for data communication between to different devices (generic computer(linux,windows) and the ESP32S2) and maybe I will introduce it to my esp32-database, I can not only use one instance because I need more then one document to handle the data. And I can not say how big the data is.
The memory fragmentation is not only caused by the json code but other code/liberys do that to so I try to minimize it.

As far as I tried. It is not possible to do what I want because "JsonDocument is private within this context" which means I can not convert, assing an new value/object and pointers do not work very well for me either. It seems I have to scrap the idea.

djpx · June 11, 2022, 6:14pm

I had a diffrend problem the memory goes puff, was no concern at this step since I was trying to get a way to make a dynamic assignment which failed.

my last code I used was this:
main.cpp:

#include <ArduinoJson.h>
#include <Arduino.h>
#include "UseJSON.h"

void setup()
{
  delay(2000);
  Serial.begin(115200);
  UseJSON useJson;
  char json[] = "{\"sensor\":\"gps\",\"time\":1351824120,\"data\":[48.756080,2.302038]}";
  JsonDocument* document = useJson.stringToJson(json);
  String sensor = (*document)["sensor"];
  Serial.println("sensor:");
  Serial.println(sensor);
}

void loop()
{

}

#include <Arduino.h>
#include <ArduinoJson.h>
#pragma once



class UseJSON
{
public:
  UseJSON();
  ~UseJSON();
  JsonDocument* stringToJson(String json,bool useDynamicDocument=false);
  String jsonToString(JsonDocument& document);
  JsonDocument* document;
  String* jsonString;
  

private:
  
    
};

#include "UseJSON.h"

UseJSON::UseJSON()
{
}

UseJSON::~UseJSON()
{
}

String UseJSON::jsonToString(JsonDocument &document)
{
    char *json;
    ArduinoJson::serializeJson(document, *jsonString);
    return *jsonString;
}

JsonDocument* UseJSON::stringToJson(String jsonString, bool useDynamicDocument)
{
    const int documentSize = jsonString.length();
    if (useDynamicDocument)
    {
        const int documentSize = jsonString.length();
        document = new DynamicJsonDocument(documentSize);
    }
    else
    {
        if (documentSize <= 50)
        {
            document = new StaticJsonDocument<50>();
        }
        else if (documentSize <= 100)
        {
            document = new StaticJsonDocument<100>();
        }
        else if (documentSize <= 150)
        {
            document = new StaticJsonDocument<150>();
        }
        else if (documentSize <= 200)
        {
            document = new StaticJsonDocument<200>();
        }
        else if (documentSize <= 250)
        {
            document = new StaticJsonDocument<250>();
        }
        else if (documentSize <= 300)
        {
            document = new StaticJsonDocument<300>();
        }
        else if (documentSize <= 400)
        {
            document = new StaticJsonDocument<400>();
        }
        else if (documentSize <= 500)
        {
            document = new StaticJsonDocument<500>();
        }
        else if (documentSize <= 600)
        {
            document = new StaticJsonDocument<600>();
        }
        else if (documentSize <= 700)
        {
            document = new StaticJsonDocument<700>();
        }
        else if (documentSize <= 800)
        {
            document = new StaticJsonDocument<800>();
        }
        else if (documentSize <= 900)
        {
            document = new StaticJsonDocument<900>();
        }
        else if (documentSize <= 1024)
        {
            document = new StaticJsonDocument<1024>();
        }
        // If stringsize is too big for the StaticJsonDocument
        else
        {
            document = new DynamicJsonDocument(documentSize);
        }

        DeserializationError error = deserializeJson(*document, jsonString);

        // Test if parsing succeeds.
        if (error)
        {
            Serial.print(F("deserializeJson() failed: "));
            Serial.println(error.f_str());
            // return null;
        }
        return document;
    }
}

I tried many variations with this code but none did work for me.

J-M-L · June 11, 2022, 6:16pm

you would need to create a copy that you pass back but it's a best practice to let the caller handle memory allocation so your library should not worry about creating the jsonDocument in the first place and let that responsibility to the caller who could pass this by reference to your library

Danois90 · June 11, 2022, 6:34pm

I really do not understand, what you are trying to achieve with "UseJSON", it seems to complicate things for no reason. Since you are using an ESP board, you are not crippled by the crude memory manager used in AVR based boards, and there is no problem in using DynamicJson - even for smaller documents.

class UseJSON
{
private:
  DynamicJsonDocument json(1024);
public:
  JsonDocument* setJson(String jsonString)
  {
    json.set(jsonString);
    return &json;
  }
};

Btw. You should put "#prama once" at the very top (first line) of your header.

djpx · June 11, 2022, 6:57pm

That would destroy the reason for the class. The caller should not have to worry about the allocation and just use it without doing any extra steps. So I do not have to create in every class an extra logic layer which has to decide which allocation should be used. I prefer to have modular classes so I can use them in multiple projects. And if I change them the caller do not notice that because the "interface" stays the same.

djpx · June 11, 2022, 7:08pm

The problem why it is complicated is, because it does not work the normal way. (It is because of the way, how this lib is implemented and what I would like to do is not really intended).
A problem is for example when I understand it right on the webside that every

StaticJsonDocument<number>

Is a different class so i can not use simple pointers with them and casting does not work really well either. Thats why I use

JsonDocument

But that creates new problems ...
Since It gets to complex I might abort that and try later again.

Btw. You should put "#prama once" at the very top (first line) of your header.
What do you mean by that I already got that im my UseJSON.h a "#prama once" or do you mean something else?

J-M-L · June 11, 2022, 7:52pm

who needs to worry about the lifetime of the memory allocation of you handover a memory buffer to someone outside your class ?

djpx · June 11, 2022, 8:00pm

The topic was not about lifetime, is is about the allocation speed and the overhead (and deframentation). I am not really sure what you like to say tbh.

J-M-L · June 11, 2022, 8:22pm

The point is that if your library hands over a buffer, you loose control and don’t know when to free the buffer so you can’t. It’s up to the caller to handle the memory which means your library can actually leak memory

Danois90 · June 12, 2022, 8:56am

Memory fragmentation is not an issue on ESP, it is an issue related to AVR. When using templates in C(++) you must consider, that for each template there will be created a new type and this will increase the code size. What you are trying to do makes - in that regard - no sense at all.

template <class T> T TPow(T value) { return value * value; }

void setup()
{
  Serial.begin(9600);

  char c = TPow<char>(10);
  Serial.println(c);
  
  /*int i = TPow<int>(10);
  Serial.println(i);*/
  
  /*long l = TPow<long>(10);
  Serial.println(l);*/
  
  /*float f = TPow<float>(10);
  Serial.println(f);*/
}

void loop() {}

For each of the commented code blocks you un-comment, the function "TPow" will be included with a new type. If all blocks are uncommented the code becomes equal to:

char CPow(char value) { return value * value; }
int IPow(int value) { return value * value; }
long LPow(long value) { return value * value; }
float FPow(float value) { return value * value; }

void setup()
{
  Serial.begin(9600);

  char c = CPow(10);
  Serial.println(c);

  int i = IPow(10);
  Serial.println(i);

  long l = LPow(10);
  Serial.println(l);

  float f = FPow(10);
  Serial.println(f);
}

void loop() {}

Templates are not expanded during runtime, they are expanded during compile time (or is it during pre-processing?). Templates takes some code and duplicates it with different values or types.

Your code therefore includes ALL the of the static json document sizes you "if .. else" into the code, but only the one needed is returned, efficient? NO!

djpx · June 12, 2022, 9:11am

You could do that but it would make the wrapper usless then.

Like I said that is not a library it is just a wrapper. And this is not a finished class it is just intended to get it to work first. When it is finished the memory is managed in the instance so it frees the memory when it is not needed any more.

When I would let the caller make the memory claim, then i would need to know how mutch data I need when I write the code. But I do not know that when I write to code so it has to change dynamicly.

J-M-L · June 12, 2022, 9:17am

is subclassing an option?

djpx · June 12, 2022, 9:20am

Code size does not matter I got enough flash. Ram on the other hand could be more importend. But it is not a big problem at the moment. But creating/reserving ram is an expensive operation (time) so i do not want to claim 10x more ram as needed.

Can you please provide a link with source how the ESP32 is handeling that so I can read more about it?

djpx · June 12, 2022, 9:25am

Yes that maybe could be an option. Which class would you take to create a subclass? The JsonDocument or the StaticJsonDocument or a diffrend class.
Or do you mean a nested class?

J-M-L · June 12, 2022, 9:31am

A StaticJsonDocument is a JsonDocument that allocates its memory pool in-place, so it doesn’t rely on dynamic memory allocation.

A DynamicJsonDocument is a JsonDocument that allocates its memory pool in the heap.

seems you need a different kind of JsonDocument that would decide on the fly which one to instantiate.

it's hard to think it through though because I don't know what the wrapper is for. may be it's totally unrelated and you just need to use polymorphism and have a pointer to a JsonDocument that will be dynamically instantiated at run time

Topic		Replies	Views
StaticJsonDocument with a variable to set the size Arduino Yún	3	2448	August 7, 2022
Parsing JSON with ArduinoJson Programming Questions	5	855	May 6, 2021
ArduinoJson 7 JsonDocument and adding a JsonObject Programming Questions	4	919	August 16, 2024
ArduinoJson null after function return Programming Questions	4	951	August 2, 2022
Using JSON data to turn on an LED Programming Questions	3	1357	May 6, 2021

How can i use ArdoinoJSON with an own wrapper

Related topics