problem parsing long JSON answer

Projects Networking, Protocols, and Devices
1

Hello.
I have a problem with parsing a very long JSON answer that I want to analyze. The answer is a list of all of a certain user's SoundCloud tracks. I am using a Wemos D1 mini and the famous JSON library by bblanchon. I heavily rely on his own example sketches and when I use them with API request that only have short answers everything works fine. As soon as I try to parse the long list of tracks though, the sketch tells me "parsing failed".

Is this a common problem with longish JSON answers? Am I running out of memory? How much memory does the ESP8266 have? Is there a chance to store that huge string away somewhere else intermediately?

I used the tool to calculate the memory needed https://bblanchon.github.io/ArduinoJson/assistant/ . The API request and its very long answer can be seen here: http://api.soundcloud.com/users/229446377/tracks?client_id=JlZIsxg2hY5WnBgtn3jfS0UYCl0K8DOg . And my complete sketch that throws the error is here: Arduino Cloud .

This is the bit of code that encounters the error:

// Parse the JSON from the input string and extract the interesting values
bool readResponseContent(struct UserData* userData) {
  // Compute optimal size of the JSON buffer according to what we need to parse.
  // See https://bblanchon.github.io/ArduinoJson/assistant/

  const size_t BUFFER_SIZE = 
    JSON_ARRAY_SIZE(18)         // 18 Objects in Array bc 18 tracks
    + 18*JSON_OBJECT_SIZE(8)    // 8 items in user object
    + 18*JSON_OBJECT_SIZE(48)   // 48 items in every object
    + MAX_CONTENT_SIZE;          // additional space for strings as given at the top

  // Allocate a temporary memory pool
  DynamicJsonBuffer jsonBuffer(BUFFER_SIZE);
  Serial.println("JSON Buffer is: ");
  Serial.println(BUFFER_SIZE);

  JsonArray& root = jsonBuffer.parseArray(client);   //client response being parsed into root
  JsonArray& root_ = root;
  JsonObject& root_0 = root_[0];
  int root_0_playback_count = root_0["playback_count"];
  
  if (!root.success()) {
    Serial.println("JSON parsing failed!");
    return false;
  }

  // Here were copy the value we're interested in
  userData->playCount = root_0_playback_count;

  return true;
}

And here's what the serial monitor shows:

Connect to api.soundcloud.com
Connected
GET /users/229446377/tracks?client_id=JlZIsxg2hY5WnBgtn3jfS0UYCl0K8DOg
JSON Buffer is: 
74357
JSON parsing failed!
Disconnect
Wait 60 seconds
2

Is it that hard to post a damned link to the hardware you're using? Please read the static post at the top of the topic.

Am I running out of memory?

Probably yes.

How much memory does the ESP8266 have?

You can use about 40kB if connected to a WiFi network.

You've chosen the wrong platform for doing such tasks. Use a Raspberry Pi or something similar, it has enough resources to access web services as the soundcloud API.

3

Is it that hard to post a damned link to the hardware you're using? Please read the static post at the top of the topic.

I thought it were enough to say that it's a Wemos D1 mini. But here's a link: https://wiki.wemos.cc/products:d1:d1_mini How can I tell how much memory is too much?

4

How can I tell how much memory is too much?

Look at the JSON parsing library. If it use malloc() or new() (as it almost certainly does), check everywhere that that happens that malloc() or new() succeeded. If it fails, or when it fails, it is because you have used too much memory.

5

Thanks for your help. I don't know how to check if a library uses certain terms and wether those successfully execute. But I'm pretty sure the problem is memory, because the parsing works well for smaller JSON answers.

So I came up with a PHP Script that runs on my own web space and that does the parsing of the long JSON strings, calculates the value I need and returns it as a very compact JSON answer. My Arduino now only fetches this and everything is fine.

For anyone interested, have a look at the PHP here: http://mixedtinkerings.com/wp-content/uploads/2017/08/calcscplays.zip

6

Another option is use a streaming JSON parser for Arduino. The parser does not require the entire JSON response be in RAM. It extracts the elements you want as the JSON streams in.