Ethernet + json from URL

Hi

I am just trying to get the JSON from URL (using Arduino Uno and Ethernet shield), but Iam not able to solve the parsing. I used two different examples for get the http response, one of them from HERE (the second one is basically the same but it have most code in setup or loop and it use client.read().. )

I have ArduinoJson 5.13.5 library and problem is that even when I use the code from the link above, it end with "JSON parsing failed!". It doesnt use the client.read(); anywhere so I dont know where the whole response should be, because if I put Serial.print(client); after the JsonObject& root = jsonBuffer.parseObject(client); to find out whats in the (client) because the parseObject obviously use it, I get only one char which is 1

I tried to use the

  while (client.available()) {
    // client data available to read
    char c = client.read();
    jsonres += c;
    
  }

where the jsonres is String (so I could have the whole response in one variable) and when i print the jsonres it really does contain the whole json response, but I can not parse it becase the jsonBuffer.parseObject needs char and if i try use the toCharArray() to add the string to char and use it, the whole code just go crazy and suddenly it even doesnt connect to the server ....

Try something like this

  int index = 0;
  while (client.available()) c[index++] = client.read();
  c[index] = '\0';

where c is declared as global char* c; and parse the char array

lesept:
Try something like this

  int index = 0;

while (client.available()) c[index++] = client.read();
  c[index] = '\0';




where c is declared as global char* c; and parse the char array

I tried it exactly as you wrote it or even as:

int index = 0;
  while (client.available()){
    c[index++] = client.read();
  }
  c[index] = '\0';

but the serial just goes crazy with this code... printing S or 5 all over again like this:

15:42:11.492 -> GET /api/index.php?cid=01454&⸮⸮⸮5555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555

You said

where the jsonres is String (so I could have the whole response in one variable) and when i print the jsonres it really does contain the whole json response

so what in this exact case is the expected string?

Try:

int index = 0;
  while (client.available()){
    c[index] = client.read();  // <-- removed the ++ here /!\
    if (c[index++] == '\r') break;
  }
  c[index] = '\0';

or

int index = 0;
  while (client.available()){
    c[index] = client.read();  // <-- removed the ++ here /!\
    if (c[index] == '\r') break;
    index++;
  }
  c[index] = '\0';

The first code works better (no repeating S or 5) but still "JSON parsing failed!"

The thing is I have API in PHP that returns something like this:

{"Status":2,"Description":"Record stored","UserName":"John White"}

When I use the String varainble to store the chars like I already posted see:

while (client.available()) {
    // client data available to read
    char c = client.read();
    jsonres += c; 
  }
  Serial.println(jsonres);

I have exactly the same {"Status":2,"Description":"Record stored","UserName":"John White"} in that jsonres. But even when I use JsonObject& root = jsonBuffer.parseObject(jsonres); its still ends with "JSON parsing failed!". The first code of yours also ends with "JSON parsing failed!" --> I had to change the parsing to JsonArray& root = jsonBuffer.parseArray(c); because if I get it right the c in your code is array

You are supplying snippets and we can only guess what may be the cause of the problem. If the JSON is valid and received correctly, then the problem may be that the JSON buffer is too small to hold the entire JSON object.

I used Assistant | ArduinoJson 5 for the buffer size, so this sould be right.

The entire code is here:

#include <ArduinoJson.h>
#include <Ethernet.h>
#include <SPI.h>

EthernetClient client;

// Name address for Open Weather Map API
const char* server = "mywebpage.tld";              //I have correct domain address of my web here

// Replace with your unique URL resource
const char* resource = "/api/index.php?cid=12547&op=22";

char* c; 

const unsigned long HTTP_TIMEOUT = 10000;  // max respone time from server
const size_t MAX_CONTENT_SIZE = 512;       // max size of the HTTP response

byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};

// The type of data that we want to extract from the page
struct clientData {
  char username;
  int  state;
};

// ARDUINO entry point #1: runs once when you press reset or power the board
void setup() {
  Serial.begin(9600);
  while (!Serial) {
    ;  // wait for serial port to initialize
  }
  Serial.println("Serial ready");
  if (!Ethernet.begin(mac)) {
    Serial.println("Failed to configure Ethernet");
    return;
  }
  Serial.println("Ethernet ready");
  delay(1000);
}

// ARDUINO entry point #2: runs over and over again forever
String jsonres;
void loop() {

  

  if (connect(server)) {
    if (sendRequest(server, resource) && skipResponseHeaders()) {
      clientData clientData;
      if (readReponseContent(&clientData)) {
        printclientData(&clientData);
      }
    }
  }
  disconnect();
  wait();
}

// Open connection to the HTTP server
bool connect(const char* hostName) {
  Serial.print("Connect to ");
  Serial.println(hostName);
  bool ok = client.connect(hostName, 80);
  Serial.println(ok ? "Connected" : "Connection Failed!");
  return ok;
}

// Send the HTTP GET request to the server
bool sendRequest(const char* host, const char* resource) {
  Serial.print("GET ");
  Serial.println(resource);
  client.print("GET ");
  client.print(resource);
  client.println(" HTTP/1.1");
  client.print("Host: ");
  client.println(host);
  client.println("Connection: close");
  client.println();
  return true;
}

// Skip HTTP headers so that we are at the beginning of the response's body
bool skipResponseHeaders() {
  // HTTP headers end with an empty line
  char endOfHeaders[] = "\r\n\r\n";
  client.setTimeout(HTTP_TIMEOUT);
  bool ok = client.find(endOfHeaders);
  if (!ok) {
    Serial.println("No response or invalid response!");
  }
  return ok;
}


bool readReponseContent(struct clientData* clientData) {
  const size_t capacity = JSON_OBJECT_SIZE(3) + 70;
  DynamicJsonBuffer jsonBuffer(capacity);

   /*
   while (client.available()) {
    // client data available to read
    char c = client.read();
    jsonres += c; 
  }
  Serial.println(jsonres);
  */

  int index = 0;
  while (client.available()){
    c[index] = client.read();  // <-- removed the ++ here /!\
    if (c[index++] == '\r') break;
  }
  c[index] = '\0';

  //JsonObject& root = jsonBuffer.parseObject(jsonres);
  JsonArray& root = jsonBuffer.parseArray(c);
  
  if (!root.success()) {
    Serial.println("JSON parsing failed!");
    return false;
  }

  // Here were copy the strings we're interested in using to your struct data
  strcpy(clientData->username, root["UserName"]);
  strcpy(clientData->state, root["Status"]);


  return true;
}

// Print the data extracted from the JSON
void printclientData(const struct clientData* clientData) {
  Serial.print("State: ");
  Serial.println(clientData->state);
  Serial.print("Name: ");
  Serial.println(clientData->username);
}

// Close the connection with the HTTP server
void disconnect() {
  Serial.println("Disconnect");
  client.stop();
}

// Pause for a 2 minute
void wait() {
  Serial.println("Wait 120 seconds");
  delay(120000);
}

"char* c;" is a pointer declaration with no memory, and you use it as incoming buffer which will cause memory corruption. It should be declared as "char c[100];" which will ensure a buffer of 100 chars.

Thanks, changed to char c[100]; but unfortunately its still JSON parsing failed! :frowning:

"jsonBuffer.parseArray" will parse an array of objects whereas "jsonBuffer.parseObject" will parse a single object. The json you have supplied in this thread is an object and not an array.

EDIT: Your next problem is "strcpy(clientData->username, root["UserName"]);" and "strcpy(clientData->state, root["Status"]);" - you are copying strings into unallocated memory or integers.

Ok maybe I'm dumb but I thought when the c is array then I need to use parseArray(). Probably not, so I change it to JsonObject& root = jsonBuffer.parseObject(c); but still no change, still JSON parsing failed!. It will not get to the "strcpy(clientData->username, root["UserName"]);" part so I guess its not a big deal right now when the parsing doesnt work :frowning:

No, a json array has the form "[item, item, item]" whereas an object has the form "{field:value, field:value, field:value}" so it has nothing to do with the data type you supply. After the line "c[index] = '\0';" try to put a "Serial.println(c);" and see if the json is as it should be.

Yes i think it is, here is what it printed in serial monitor:

20:44:11.178 -> C: {"Status":2,"Description":"Record stored","UserName":"John White"}
20:44:11.279 -> JSON parsing failed!

EDIT: the code I added is:

Serial.print("C: ");
Serial.print(c);

Dunno what's wrong then but try to declare a larger json buffer by replacing "DynamicJsonBuffer jsonBuffer(capacity)" with "DynamicJsonBuffer jsonBuffer(200)".

so I tried with:

DynamicJsonBuffer jsonBuffer(69);
DynamicJsonBuffer jsonBuffer(75);
DynamicJsonBuffer jsonBuffer(85);
DynamicJsonBuffer jsonBuffer(100);
DynamicJsonBuffer jsonBuffer(120);
DynamicJsonBuffer jsonBuffer(150);
DynamicJsonBuffer jsonBuffer(180);
DynamicJsonBuffer jsonBuffer(200);
DynamicJsonBuffer jsonBuffer(230);

StaticJsonBuffer<100> jsonBuffer;
StaticJsonBuffer<200> jsonBuffer;
StaticJsonBuffer<250> jsonBuffer;

but everything is still JSON parsing failed!

EDIT:
I tried to rewrite the JSON parser to the new library version (6.10.1) which can return errors. So the print of c is exactly the same as was (which is corect json I think) but the parser returns: deserializeJson() failed with code InvalidInput

Go back to using

JsonObject& root = jsonBuffer.parseObject(json);

and declare the char * c as

const char* c[100];

lesept:
Go back to using

JsonObject& root = jsonBuffer.parseObject(json);

and declare the char * c as

const char* c[100];

I tried it but with the const char* c[100]; Iam getting error - no matching function for call to 'print(const char* [100])'

Sorry remove the * in the array declaration

Please try the following code and see if it causes any errors:

void setup()
{
  Serial.begin(9600);
  char* jsonStr = "{\"Status\":2,\"Description\":\"Record stored\",\"UserName\":\"John White\"}";
  DynamicJsonBuffer jsonBuffer(200);
  JsonObject& jsonObj = jsonBuffer.parseObject(jsonStr);
  if (jsonObj.success()) Serial.println("JSON success");
  else Serial.println("JSON error");
}

If it works, you problem may be some illegal characters in the JSON you receive.

Danois90:
Please try the following code and see if it causes any errors:

void setup()

{
  Serial.begin(9600);
  char* jsonStr = "{"Status":2,"Description":"Record stored","UserName":"John White"}";
  DynamicJsonBuffer jsonBuffer(200);
  JsonObject& jsonObj = jsonBuffer.parseObject(jsonStr);
  if (jsonObj.success()) Serial.println("JSON success");
  else Serial.println("JSON error");
}




If it works, you problem may be some illegal characters in the JSON you receive.

This one works... but how can I find out what illegal character is there? If Iam looking on the webpage sourcecode there is just the json, exactly the same one that is printed in the serial monitor when I print the variable inside which the json should be... so I can not see any illegal characters

EDIT: seems you are right, I tried to parse this example https://jsonplaceholder.typicode.com/todos/1 and it worked just fine. So there has to eb something wrong with the encoding of I dont know, I will try to find out why the hell my response is not parsed....

EDIT2: ok this is very strange, I just make very simple copy of my api (it only returns the json and the values are set into the variables instead of obtained from the database). I uploaded this simple version to the another website (because the first one runs only on our local network so its not accessible from outside but the acruino can access it...) and tried to validate the json from URL at this page https://jsonformatter.curiousconcept.com/ - but Ive got error "Invalid encoding, expecting UTF-8, UTF-16 or UTF-32.". Strange thing is the arduino is able to parse it just fine. So from the local website.