ESP8266 JSON too large & Crash

I'm Pretty new the Arduino scene and I'm using a Wemos d1 Pro Mini and I'm trying to parse a JSON URL(Just printing the whole string to serial for now) but its way to big and my board just resets.

JSON:https://poloniex.com/public?command=returnTicker

I know the code works because I can get this https://poloniex.com/public?command=returnOrderBook&currencyPair=BTC_NXT&depth=50

While doing a google search I found GitHub - squix78/json-streaming-parser: Arduino library for parsing potentially huge json streams on devices with scarce memory but I do not understand how to get it to work.

Essentially what I want it to do is pull

"BTC_BURST":{"id":15,"last":"0.00000675","lowestAsk":"0.00000675","highestBid":"0.00000674","percentChange":"0.10655737","baseVolume":"2452.31027933","quoteVolume":"372519946.72358555","isFrozen":"0","high24hr":"0.00000755","low24hr":"0.00000550"}

From https://poloniex.com/public?command=returnTicker

Any help would be appreciated.

Here is my code so far, it uses the ESP8266 Wifi manager.

#include <ESP8266HTTPClient.h>
#include <ESP8266WiFi.h>
#include <DNSServer.h>
#include <ESP8266WebServer.h>
#include <WiFiManager.h>
#include <LiquidCrystal_I2C.h>
#include <Wire.h> 
#include <ArduinoJson.h>
#define LCD_ROWS 2
#define LCD_COLS 16

LiquidCrystal_I2C lcd(0x3F, LCD_COLS, LCD_ROWS);
WiFiManager wifiManager;
char apname[] = "ssid";
//char appass[] = "pass";



void configModeCallback (WiFiManager *myWiFiManager) {
  lcd.home();
  lcd.clear();
  lcd.print("Now Entering");
  lcd.setCursor(0, 1);
  lcd.print("Setup Mode");
  delay(2000);
  lcd.home();
  lcd.clear();
  lcd.print("Pls Conn To AP");
  lcd.setCursor(0, 1);
  lcd.print(apname);
}


void setup() {
  Serial.begin(115200);
  Serial.println();
  lcd.begin(4,5);
  lcd.backlight();
  startup();
  thankyou();
  lcd.home();
  lcd.clear();
  lcd.print("Connecting");   
  lcd.setCursor(0, 1);
  lcd.print("To Local WiFi");
  wifiManager.setTimeout(180);
  wifiManager.setAPCallback(configModeCallback);
  wifiManager.autoConnect(apname); //pass would go here   
  lcd.clear();
  lcd.print("Connected!");
  delay(2000); 
}

void resetesp(){
    wifiManager.resetSettings();
}

void startup(){
  lcd.clear();
  lcd.home();
  lcd.print("Ticker Test");
  lcd.setCursor(0, 1);
  lcd.print("By Draknoid");
  delay(2000);
}

void thankyou(){
  lcd.clear();
  lcd.home();
  lcd.print("Thank You!");
  lcd.setCursor(0, 1);
  lcd.print("Your Name Here");
  delay(2000);
}

void loop() {
 
if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status
 
    HTTPClient http;  //Declare an object of class HTTPClient
 
    http.begin("https://poloniex.com/public?command=returnTicker","83:7D:87:4B:80:8B:B9:26:33:C0:5A:DC:30:18:58:D9:69:14:D1:4F");  //Specify request destination
    int httpCode = http.GET();//Send the request
 
    if (httpCode > 0) { //Check the returning code
 
      String payload = http.getString();   //Get the request response payload
      Serial.println(payload);                     //Print the response payload
 
    }
 
    http.end();   //Close connection
 
  }
 
  delay(50000);    //Send a request every 30 seconds
 
}

This is the getString code

String HTTPClient::getString(void)
{
    StreamString sstring;

    if(_size) {
        // try to reserve needed memmory
        if(!sstring.reserve((_size + 1))) {
            DEBUG_HTTPCLIENT("[HTTP-Client][getString] not enough memory to reserve a string! need: %d\n", (_size + 1));
            return "";
        }
    }

    writeToStream(&sstring);
    return sstring;
}

The debug function is a conditional define

#ifdef DEBUG_ESP_HTTP_CLIENT
#ifdef DEBUG_ESP_PORT
#define DEBUG_HTTPCLIENT(...) DEBUG_ESP_PORT.printf( __VA_ARGS__ )
#endif
#endif

#ifndef DEBUG_HTTPCLIENT
#define DEBUG_HTTPCLIENT(...)
#endif

You need to define DEBUG_ESP_HTTP_CLIENT and DEBUG_ESP_PORT to see a possible memory error message

I'm not sure how to use that getstring code but this is what my serial monitor is showing me without the debug code

ets Jan  8 2013,rst cause:4, boot mode:(3,6)

wdt reset
load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v60000318
~ld

and trying to use the string code to the best of my knowledge but it gives me this

'StreamString' was not declared in this scope

I still can't get my head around that JSONstreamimg parser, can anyone point me in the right direction I'm pretty new to C# and the Arduino scene

I'm pretty new to it all too, but had a similar thing happening. I notice it looks like a WDT reset (ie Watch Dog Timer). This is a long shot, but try changing from 80MHz to 160MHz. It solved a few WDT reset issues I was having.

It's in Tools->CPU Frequency.

NOTE: I'm pretty new too, so this is a long shot.

Yeah sadly it did not work, I wonder if there would be a way to just get the information I'm looking for from the JSON and nothing else. Or download in chunks until I get the information I'm looking for.

Something is taking too long. How long does setup() take? Maybe all the writing on the lcd takes quite a long time? Put some yield() or delay(10) after each time you're spending time writing on the lcd? What did the serial say befor it showed the wdt error? That might tell you where it got in the code befor the wdt kicked.

Jimmy

See if I call https://poloniex.com/public?command=returnOrderBook&currencyPair=BTC_NXT&depth=50 it returns

 {"asks":[["0.00002802",481.42162103],["0.00002835",3.67190632],["0.00002836",20638.41032792],["0.00002837",23554.66373567],["0.00002839",3.59844977],["0.00002840",770.31760716],["0.00002841",3.59585536],["0.00002842",10455.94671577],["0.00002843",76285.59326469],["0.00002844",3.59197075],["0.00002845",3.59067775],["0.00002847",3.58809453],["0.00002848",95.49669884],["0.00002849",3.58551502],["0.00002851",17624.07576511],["0.00002852",1854.12777032],["0.00002853",98.24192716],["0.00002854",12806.866],["0.00002855",1922.6496667],["0.00002856",40876.44751588],["0.00002857",3.57523398],["0.00002860",4154.29204263],["0.00002861",30631.76343004],["0.00002862",3.57011553],["0.00002864",3.5675618],["0.00002865",953.87058765]

I had to cut it
and that's only a few results but the one I need to get information from https://poloniex.com/public?command=returnTicker

{"BTC_BCN":{"id":7,"last":"0.00000143","lowestAsk":"0.00000143","highestBid":"0.00000142","percentChange":"-0.12269938","baseVolume":"3418.74757011","quoteVolume":"2273910010.38433123","isFrozen":"0","high24hr":"0.00000173","low24hr":"0.00000123"},"BTC_BELA":{"id":8,"last":"0.00008499","lowestAsk":"0.00008499","highestBid":"0.00008438","percentChange":"-0.05134501","baseVolume":"131.17294577","quoteVolume":"1497528.54450695","isFrozen":"0","high24hr":"0.00009180","low24hr":"0.00008250"},"BTC_BLK":{"id":10,"last":"0.00008900","lowestAsk":"0.00008900","highestBid":"0.00008899","percentChange":"-0.09909909","baseVolume":"120.83794517","quoteVolume":"1271661.05104403","isFrozen":"0","high24hr":"0.00010642","low24hr":"0.00008178"},"BTC_BTCD":{"id":12,"last":"0.01806964","lowestAsk":"0.01806964","highestBid":"0.01800103","percentChange":"0.12935179","baseVolume":"249.54629766","quoteVolume":"13174.37478356","isFrozen":"0","high24hr":"0.02225000","low24hr":"0.01557937"},"BTC_BTM":{"id":13,"last":"0.00042119","lowestAsk":"0.00042151","highestBid":"0.00042060","percentChange":"-0.11484952","baseVolume":"121.55451045","quoteVolume":"254616.37514906","isFrozen":"0","high24hr":"0.00052598","low24hr":"0.00040100"},"BTC_BTS":{"id":14,"last":"0.00003978","lowestAsk":"0.00003978","highestBid":"0.00003976","percentChange":"-0.11066398","baseVolume":"4635.25657628","quoteVolume":"118651211.27364478","isFrozen":"0","high24hr":"0.00004543","low24hr":"0.00003334"},"BTC_BURST":{"id":15,"last":"0.00000558","lowestAsk":"0.00000555","highestBid":"0.00000551","percentChange":"-0.12676056","baseVolume":"967.54009879","quoteVolume":"160468336.36221683","isFrozen":"0","high24hr":"0.00000683","low24hr":"0.00000500"},"BTC_CLAM":{"id":20,"last":"0.00205975","lowestAsk":"0.00205976","highestBid":"0.00205975","percentChange":"-0.03978387","baseVolume":"1440.83821718","quoteVolume":"637767.71956716","isFrozen":"0","high24hr":"0.00263100","low24hr":"0.00174186"}

I had to cut more than half to post
and that's about 500 results and I know it's too big, all I need is BTC_BURST from there

I'm pretty new to C# and the Arduino scene

Since the Arduino is not programmed using C#, does that matter?

you realized you cannot contain the whole JSON object, so you are half way there.

I would try to parse it in chunks, and you should not use String class. Create a buffer big enough to store each property:

char jsonBuffer[128]; // verify this

read the stream until you get [128] characters or see a closing brace (this is all pseudo):

while(Serial.peek() != '}';
{
  addToArray();
}

look for the key you want to find after you consume each chunk... and use strtok to get your values:

if(strstr(jsonBuffer, "BTC_BCN"))
{
  strtok(...
  ...
}

Ok so I'm not home right now so I cannot try it but based off the ESP8266HTTPCLIENT stream example and what bulldogLowell wrote I'm guessing something like this should work just to print out the entire thing to serial.

void loop() {
 
	if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status

		HTTPClient http;  //Declare an object of class HTTPClient
		http.begin("https://poloniex.com/public?command=returnTicker","83:7D:87:4B:80:8B:B9:26:33:C0:5A:DC:30:18:58:D9:69:14:D1:4F");  //Specify request destination
		int httpCode = http.GET();//Send the request

		if(httpCode == HTTP_CODE_OK){

			int len = http.getSize();// get lenght of document (is -1 when Server sends no Content-Length header)
			// create buffer for read
			char jsonBuffer[128]; // verify this
			// get tcp stream
			WiFiClient * stream = http.getStreamPtr();

			// read all data from server
			while(http.connected() && (len > 0 || len == -1)) {
				// get available data size
				size_t size = stream->available();

				if(size) {
					// read up to 128 byte
					int c = stream->readBytes(jsonBuffer, ((size > sizeof(jsonBuffer)) ? sizeof(jsonBuffer) : size));
					// write it to Serial
					serial.println(F(jsonBuffer, c));

					if(len > 0) {
						len -= c;
					}
				}
			}
		}
	}
}

Is there a way to use something like serial.peek() but for the wifi stream to see if there is a }
Code is untested and can only test once I get home later. Thanks again for all the help

Ok so this code works and it prints to the serial monitor

#include <ESP8266HTTPClient.h>
#include <ESP8266WiFi.h>
#include <DNSServer.h>
#include <ESP8266WebServer.h>
#include <WiFiManager.h>
#include <LiquidCrystal_I2C.h>
#include <Wire.h> 
#include <ArduinoJson.h>
#define LCD_ROWS 2
#define LCD_COLS 16

LiquidCrystal_I2C lcd(0x3F, LCD_COLS, LCD_ROWS);
WiFiManager wifiManager;
char apname[] = "ssid";
//char appass[] = "pass";



void configModeCallback (WiFiManager *myWiFiManager) {
  lcd.home();
  lcd.clear();
  lcd.print("Now Entering");
  lcd.setCursor(0, 1);
  lcd.print("Setup Mode");
  delay(2000);
  lcd.home();
  lcd.clear();
  lcd.print("Pls Conn To AP");
  lcd.setCursor(0, 1);
  lcd.print(apname);
}


void setup() {
  Serial.begin(115200);
  Serial.println();
  lcd.begin(4,5);
  lcd.backlight();
  startup();
  thankyou();
  lcd.home();
  lcd.clear();
  lcd.print("Connecting");   
  lcd.setCursor(0, 1);
  lcd.print("To Local WiFi");
  wifiManager.setTimeout(180);
  wifiManager.setAPCallback(configModeCallback);
  wifiManager.autoConnect(apname); //pass would go here   
  lcd.clear();
  lcd.print("Connected!");
  delay(2000); 
}

void resetesp(){
    wifiManager.resetSettings();
}

void startup(){
  lcd.clear();
  lcd.home();
  lcd.print("Ticker Test");
  lcd.setCursor(0, 1);
  lcd.print("By Draknoid");
  delay(2000);
}

void thankyou(){
  lcd.clear();
  lcd.home();
  lcd.print("Thank You!");
  lcd.setCursor(0, 1);
  lcd.print("Your Name Here");
  delay(2000);
}

void loop() {
 
  if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status

    HTTPClient http;  //Declare an object of class HTTPClient
    http.begin("https://poloniex.com/public?command=returnTicker","83:7D:87:4B:80:8B:B9:26:33:C0:5A:DC:30:18:58:D9:69:14:D1:4F");  //Specify request destination
    int httpCode = http.GET();//Send the request

    if(httpCode == HTTP_CODE_OK){

      int len = http.getSize();// get lenght of document (is -1 when Server sends no Content-Length header)
      // create buffer for read
      char jsonBuffer[256]; // verify this
      // get tcp stream
      WiFiClient * stream = http.getStreamPtr();

      // read all data from server
      while(http.connected() && (len > 0 || len == -1)) {
        // get available data size
        size_t size = stream->available();

        if(size) {
          // read up to 256 byte
          int c = stream->readBytes(jsonBuffer, ((size > sizeof(jsonBuffer)) ? sizeof(jsonBuffer) : size));
          // write it to Serial
          Serial.println(jsonBuffer);

          if(len > 0) {
            len -= c;
          }
        }
      }
    }
  }
  delay(10000);
}

But if i were to use something like this.

if(strstr(jsonBuffer, "BTC_BCN"))
{
  strtok(...
  ...
}

And "BTC_BCN" is near the end of the chunk the rest would be in the next chunk, so how would I get those? Or would there be a way to divide the chunks by either 256 bytes or } whichever comes first ?

And "BTC_BCN" is near the end of the chunk the rest would be in the next chunk, so how would I get those?

You also have the problem where BTC_BCN might span "chunks", too.

What you might want to do is look for 'B' in the jsonBuffer. If it is found, the stuff before the B is junk, so look for the marker that indicates the end of the BTC_BCN object. If you don't find it, move the data from the B to the end of jsonBuffer to another array. Then, read the next "chunk". Find the marker that indicates the end of the BTC_BCN object. Append the data up to the end marker to the other array.

Then, parse that array to get the useful data, if it contains "BTC_BCN".

If the 'B' and the end marker are in one chunk, and that chunk contains "BTC_BCN", you don't need to copy anything.

It might be easier to call a different script on a different server. That script would get the data from poloniex.com, do the parsing, and just send back the interesting data.

PaulS:
It might be easier to call a different script on a different server. That script would get the data from poloniex.com, do the parsing, and just send back the interesting data.

very good advice!

yes, maybe even a micro-service like hook.io which is free for developers. Use a webhook that parses peloniex.com and sends back a much simpler JSON object containing only the key-value pairs you are interested in.

the problem with using some other service to parse the JSON would be that it would cost money. The ticker will also make a request every 5 to 10 sec so the max request it would make per day would be 17280 and that would be about 5.1M per month and I plan on having a few devices. So I would need to do it using the ESP8266. Would it be too taxing to have the buffer 1 byte big and see if it contains a } and if not then send it to another array ?

Draknoid:
the problem with using some other service to parse the JSON would be that it would cost money. The ticker will also make a request every 5 to 10 sec so the max request it would make per day would be 17280 and that would be about 5.1M per month and I plan on having a few devices. So I would need to do it using the ESP8266. Would it be too taxing to have the buffer 1 byte big and see if it contains a } and if not then send it to another array ?

well, you will probably be rate limited by the website anyways...

back to parsing...

PaulS gave you good advice...

so for example I would have a for loop that would loop up to jsonbuffer's size and every time looking at the next char in the array ? because if i use something like

if(strstr(jsonBuffer, "}"))
{
  strtok(...
  ...
}

how would I then move everything that came before } to an array and send the rest to a new array ? Like how would I know were I am in the array ? With a for loop I could just increment an int and use that to figure out were the } is I'm guessing.

This way I would be able to not only get BTC_BURST but if I wanted I could easily get BTC_DGE

Draknoid:
so for example I would have a for loop that would loop up to jsonbuffer's size and every time looking at the next char in the array ? because if i use something like

if(strstr(jsonBuffer, "}"))

{
  strtok(...
  ...
}


how would I then move everything that came before } to an array and send the rest to a new array ? Like how would I know were I am in the array ? With a for loop I could just increment an int and use that to figure out were the } is I'm guessing.

This way I would be able to not only get BTC_BURST but if I wanted I could easily get BTC_DGE

make a buffer that is big enough contain your search key value, rotate the oldest char out as you progress, like a circular buffer. Keep checking the contents of the buffer against the key value. Once you find it, you can start to parse until you get the trailing curly brace....

Ok I'm a little further ahead now. I can more or less get it to sepperate after every }

void loop() {
 
  if (WiFi.status() == WL_CONNECTED) { //Check WiFi connection status

    HTTPClient http;  //Declare an object of class HTTPClient
    http.begin("https://poloniex.com/public?command=returnTicker","83:7D:87:4B:80:8B:B9:26:33:C0:5A:DC:30:18:58:D9:69:14:D1:4F");  //Specify request destination
    int httpCode = http.GET();//Send the request

    if(httpCode == HTTP_CODE_OK){

      int len = http.getSize();// get lenght of document (is -1 when Server sends no Content-Length header)
      // create buffer for read
      int currentPosition = 0;
      // get tcp stream
      WiFiClient * stream = http.getStreamPtr();

      // read all data from server
      while(http.connected() && (len > 0 || len == -1)) {
        // get available data size
        size_t size = stream->available();
        char jsonBuffer[128]; // verify this
        char current[265];
        if(size) {
          // read up to 128 byte
          int c = stream->readBytes(jsonBuffer, ((size > sizeof(jsonBuffer)) ? sizeof(jsonBuffer): size));
          int targetIndex = 0;

           for (int x = 0; x < sizeof(jsonBuffer); x++)
            {
              current[currentPosition] = jsonBuffer[targetIndex];

              if(jsonBuffer[targetIndex] == '}')
              {
                currentPosition = 0;
                Serial.println(current);
              }

              targetIndex++;
              currentPosition++;

            }

          if(len > 0)
          {
            len -= c;
          }
        }
      }
    }
  }
  delay(10000);
}

now the problem is that sometimes current[] will have different length varying from 230-264
now lets say i print current[] and its filled up to 264 and the next time i print it its filled to 250, once it prints it will show everything up to 250 but the rest will be old data from when it printed 264.(not sure exactly how to explain this). This is what is printed in my serial monitor.

3e64
{"BTC_BCN":{"id":7,"last":"0.00000135","lowestAsk":"0.00000136","highestBid":"0.00000135","percentChange":"-0.07534246","baseVolume":"1236.69275173","quoteVolume":"895466630.89049470","isFrozen":"0","high24hr":"0.00000146","low24hr":"0.00000129"}⸮
⸮?aseVolume":"1236.69275173","quoteVolume":"895466630.89049470","isFrozen":"0","high24hr":"0.00000146","low24hr":"0.00000129"},"BT/⸮?⸮,⸮?⸮&⸮?
3,"BTC_BELA":{"id":8,"last":"0.00009190","lowestAsk":"0.00009223","highestBid":"0.00009190","percentChange":"-0.08511697","baseVolume":"444.19989877","quoteVolume":"4640141.84014816","isFrozen":"0","high24hr":"0.00010559","low24hr":"0.00008525"}00129"}⸮
⸮?":"444.19989877","quoteVolume":"4640141.84014816","isFrozen":"0","high24hr":"0.00010559","low24hr":"0.00008525"},"BTC_BLK":{"id"/⸮?⸮,⸮?⸮&⸮?
3,"BTC_BLK":{"id":10,"last":"0.00009370","lowestAsk":"0.00009386","highestBid":"0.00009370","percentChange":"0.00106837","baseVolume":"95.44391928","quoteVolume":"1001459.90780569","isFrozen":"0","high24hr":"0.00009999","low24hr":"0.00009126"}"}00129"}⸮
⸮?8","quoteVolume":"1001459.90780569","isFrozen":"0","high24hr":"0.00009999","low24hr":"0.00009126"},"BTC_BTCD":{"id":12,"last":"0/⸮?⸮,⸮?⸮&⸮?
3,"BTC_BTCD":{"id":12,"last":"0.01847330","lowestAsk":"0.01864497","highestBid":"0.01847334","percentChange":"0.00929125","baseVolume":"64.37905346","quoteVolume":"3499.68189094","isFrozen":"0","high24hr":"0.01939900","low24hr":"0.ange":"0.00929125","baseV01752204"},"BTC_BTM":{"id":13,"01752204"}0043602","lowestAsk":"0.00043640","highestBid":"0.00043604","percentChange":"-0.0741102/⸮?⸮,⸮?⸮&⸮?
3,"BTC_BTM":{"id":13,"01752204"}01847330","lowestAsk":"0.01864497","highestBid":"0.01847334","percentChange":"0.00929125","baseVolume":"64.37905346","quoteVolume":"3499.68189094","isFrozen":"0","high24hr":"0.01939900","low24hr":"0.ange":"0.00929125","baseV01752204"},"BTC_BTM":{"id":13,"01752204"}0043602","lowestAsk":"0.00043640","highestBid":"0.00043604","percentChange":"-0.0741102/⸮?⸮,⸮?⸮&⸮?
30043602","lowestAsk":"0.00043640","highestBid":"0.00043604","percentChange":"-0.07411025","baseVolume":"95.56644553","quoteVolume":"214876.44464827","isFrozen":"0","high24hr":"0.00049390","low24hr":"0.00039026"}9900","low24hr":"0.ange":"0.00929125","baseV5","baseVolume":"95.56644553","quoteVolume":"214876.44464827","isFrozen":"0","high24hr":"0.00049390","low24hr":"0.00039026"},"BT/⸮?⸮,⸮?⸮&⸮?

I have no idea what 3e64 is and where it comes from and before every new line there is a # and at the end there is junk.