Assigning data from webpage to variables

Projects General Guidance
1

Hey everyone!
I have recently been working on a project where I receive live weather data using an API. I want to create a home weather 'clock'. Right now I am focusing on temperature data. I get the data from a website, using the esp8266. So far I have got the data, and I can 'write' it to the serial monitor. I want to be able to save the temperature data as a variable in my Arduino code, which is continuously updating as the esp8266 receives more data.

One problem I have is that the API separates the temperature data into a separate webpage, with just the temperature visible in the top left-hand corner. However, when I 'get' the data into the arduino, the result is a jumbled mess full of random sentences, then at the bottom of the package
is the data that I need.

Here is an example of the data package that gets printed to the serial monitor.

HTTP/1.1 200 OK
Date: Thu, 31 Mar 2022 22:49:13 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Status: 200 OK
X-Frame-Options: SAMEORIGIN
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.117807
X-Powered-By: Phusion Passenger
Server: nginx + Phusion Passenger

2
37
0

The '37' is the only data that I need. I have no idea what anything else means, or even where it came from, because the API only shows the 37.

So, please could someone help me assign just the temperature data to a variable that can be used in my code? Thank you so much in advance. My code is below.

#include <ESP8266WiFi.h>
#include <Wire.h>  // This library is already built in to the Arduino IDE



const char* ssid = "***********"; // put router name
const char* password = "*************";// put password 
 
const char* host = "api.thingspeak.com";



 
void setup() {
  Serial.begin(115200);
 
 // We start by connecting to a WiFi network
 
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  
  WiFi.begin(ssid, password); 
  
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
 
  Serial.println("");
  Serial.println("WiFi connected");  
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}
 
int value = 0;




 
void loop() {

 
 
  Serial.print("connecting to ");
  Serial.println(host);
  
  // Use WiFiClient class to create TCP connections
  WiFiClient client;
  const int httpPort = 80;
  if (!client.connect(host, httpPort)) {
    Serial.println("connection failed");
   
  }
  
  //now create a URL for the request
  String url = "/apps/thinghttp/send_request?api_key=************";
  Serial.print("Requesting URL: ");
  Serial.println(url);
  // This will send the request to the server
  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" + 
               "Connection: close\r\n\r\n");
  delay(500);
  
  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
   char c = client.read();
   Serial.write(c);
  }
   

  Serial.println();
  Serial.println("closing connection");





  delay(2000);
}
2

if the format of the reply is always the same you could

  1. read each line into a String
  2. check the line starts with a numeric character
  3. if it does use the String.toInt() method to convert to an integer

the second integer converted is the temperature

3

hi Horace,
thanks so much for helping me.
Do you know how I would go about reading each line of the string? I am a beginner, so despite some research, I cant seem to find how to do that. Also, would I use the startswith function to check if it starts with a number?

I noticed that the data i need is always at the very end of the package, so maybe i could use the endswith function to get the number? just a thought.
Thanks so much for your help, i really appreciate it.

4

You may find it easier if you request the data in a different format. I see from the Thingspeak docs that you can get JSON for example.

1 Like
5

an example of how to read a complete line and extract an integer is

void setup() {
 Serial.begin(115200);
}

void loop() {
 if(Serial.available()) {
  String s=Serial.readStringUntil('\n');
  Serial.println("string read " + s);
  if(s[0]>='0' && s[0]<='9') {
     Serial.print("   integer ");
     int i=s.toInt();
     Serial.println(i);
  }
}
}

it

  1. String s=Serial.readStringUntil('\n'); reads a line of text from Serial (your would use client)
  2. if(s[0]>='0' && s[0]<='9') { tests if it start with a numeric character
  3. if it does int i=s.toInt(); converts the String to an int

a run gives

string read test 1
string read test 2
string read 23
   integer 23
string read 567
   integer 567
string read 7890
   integer 7890

first couple of lines start with none numeric characters so are ignored
the rest contain valid integers

the suggestion from @wildbill sounds a good alternative - you could search the JSON for the appropriate string and extract the integer

1 Like
6

Wildbill, thanks for your suggestion. I had previously tried to use a JSON API but for some reason, I could not get the Arduino library working. Instead, I tried using a different API called openweathermap. This turned out to be very helpful, because each data packet had 'titles' before each piece of data, for example before the temperature, there would be a 'temp:'. This allowed me to parse the packets and extract pieces of data very easily. Here is the code I used to do this.

while(client.available()){
    String line = client.readStringUntil('\r');
int start_loc= find_text("mp\":",line,0);
int end_loc= find_text(",\"feels",line,0);
if (start_loc>0 && end_loc>0){
  Serial.println("temp:  ");
for (int i=start_loc+4;i<end_loc;i++)
{
Serial.print(line[i]);
}

Horace, thank you so much for taking the time to help me with this. Although I already found a solution to my problem, I will definitely try to implement your solution as well, as a more concise method of parsing the data.

Now that I have separated the data into a standalone figure in the serial monitor, I am pondering how I can then save it as a variable in my Arduino code, which can be used to manipulate different outputs. For example, I would like to save the value for temperature as a variable and have that figure cause a servo to turn if above a certain value in the serial monitor.

I tried to do this by using the value of i. As seen in the second last line of code, and the fourth last, this is what I used to output the temperature value. However, I don't know if this is the best way to do it. Also, I can't use the value of I outside the two curly brackets after the for statement, which might make it harder.

Thank you so much in advance, and also thanks for your help so far!

7

not sure where the find_text() method comes from for finding patterns in a String
why not use String.indexOf();
e.g.

void setup() {
    Serial.begin(115200);
    // assume a line of text containing a number
    String line = "xxxxmp\":37,\"feelszzzzzz";
    Serial.println(line);
    int start_loc= line.indexOf("mp\":");
   int end_loc=line.indexOf(",\"feels");
   if (start_loc>0 && end_loc>0){
      // get numeric text
      String s=line.substring(start_loc+4,end_loc);
      Serial.println("substring " + s);
      Serial.print("temp:  ");
      int i=s.toInt();  // convert it to an int
      Serial.println(i);
  }
}

void loop() {}

the above

  1. finds the patterns using String.indexOf()
  2. extracts the numeric characters using String.substring()
  3. then converts s to an integer using String.toint()
1 Like
8

not sure if this source has weatherdata from your region

9

Horace, thank you so much for taking the time to help me! You're amazing. Your solution absolutely worked, and I can't wait to implement the code for other data points such as the chance of rain.
Beef

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.