Trouble with TextFinder and ESP8266

Projects Networking, Protocols, and Devices
1

I’m way over my head here and don’t see any easy way out – perhaps someone else can point me in the right direction.

Until recently, I’ve been using Textfinder to scape information off an Environment Canada weather api and it’s worked perfectly for years. But recently, the api has demanded a fingerprint, and after much anguish, I managed to connect again using:

#include <WiFiClientSecure.h>
#include <ESP8266WebServer.h>

But my Textfinder routines no longer seem to read the data as it comes in. This one USED to work great for reading temperature and wind speed

Serial.print ("connecting to ");
char* server = "weather.gc.ca"; // first part of the url
Serial.println(server);
if (!client.connect(server, httpsPort)) {
Serial.println("connection failed");
return;
}

if (client.verify(fingerprint, server)) {
Serial.println("certificate matches");
} else {
// Serial.println("certificate doesn't match.. I don't know what this means ");
// Serial.println(" ");
}

String url = "/wxlink/site_js/s0000458_e.js"; // the second part of the url
// Serial.print("requesting URL: ");
// Serial.println(url);

client.print(String("GET ") + url + " HTTP/1.1\r\n" +
"Host: " + server + "\r\n" +
"User-Agent: BuildFailureDetectorESP8266\r\n" +
"Connection: close\r\n\r\n");

//Serial.println("Starting Connection to EnviroCanada");

if (client.connected()) {
// String line = client.readStringUntil('\n');

// if (line == "\r") {

if (finder.find("obTemperature") ) // this has to be very exact.
{
int temperature = finder.getValue();
// Serial.print("Temperature is "); //
// Serial.println(temperature);

}
else
Serial.print("Could not find Temperature field");

if (finder.find("obWindSpeed") )
{
WindSpeed = finder.getValue();
// Serial.print("WindSpeed is "); //
// Serial.println( WindSpeed );
}
else
Serial.print("Could not find WindSpeed field");

}

client.stop(); //Disconnect for next reload
client.flush();//Serial.println(line); // prints the whole page
Serial.println( " ");
Serial.print(" Temperature is "); //
Serial.println(Temperature);
Serial.print(" WindSpeed is "); //
Serial.println( WindSpeed );
Serial.println( " ");
Serial.println("Ending Connection to EnviroCanada");
Serial.println( " ");

But no longer works at all .
I thought this would be a matter of changing

WiFiClient client;
TextFinder finder(client);

To
WiFiClient httpsClient;
TextFinder finder(httpsClient);

and change all the other client references to httpsClient...

But it makes no difference and makes me think I’ve badly misunderstood the way Textfinder and client works

Anyone have any ideas?

Thanks David

2

And where is this "TextFinder" that you are using? Link to where you got the code from. Understanding this code will be key to solving your problem.

Also, have you tried printing out the string that you're searching, to make sure that it actually has the content you expect? That would be a good first step to understanding this; there may be some subtle difference in the formatting or content of the string that's causing problems.

Also, you posted a question about your project in the section for questions about installation/troubleshooting for the IDE or the board itself, despite the fact that it says at the top of the section not to do this. I have asked the moderators to move you post to the appropriate section.

3

I’m way over my head here and don’t see any easy way out – perhaps someone else can point me in the right direction.

Until recently, I’ve been using Textfinder to scape information off an Environment Canada weather api and it’s worked perfectly for years. But recently, the api has demanded a fingerprint, and after much anguish, I managed to connect again using:

#include <WiFiClientSecure.h>
#include <ESP8266WebServer.h>

But my Textfinder routines no longer seem to read the data as it comes in. This one USED to work great for reading temperature and wind speed

--------------------------------------------

Serial.print   ("connecting to ");
 char* server = "weather.gc.ca";   // first part of the url 
 Serial.println(server);
 if (!client.connect(server, httpsPort)) { 
   Serial.println("connection failed");
   return;
 }

 if (client.verify(fingerprint, server)) {
   Serial.println("certificate matches");
 } else {
  // Serial.println("certificate doesn't match.. I don't know what this means ");
  // Serial.println("  ");
 }

 String url = "/wxlink/site_js/s0000458_e.js";   // the second part of the url 
  // Serial.print("requesting URL: ");
  // Serial.println(url);

   client.print(String("GET ") + url + " HTTP/1.1\r\n" +
              "Host: " + server + "\r\n" +
              "User-Agent: BuildFailureDetectorESP8266\r\n" +
              "Connection: close\r\n\r\n");

 //Serial.println("Starting Connection to EnviroCanada");

if (client.connected()) {
    // String line = client.readStringUntil('\n');
       
   //  if (line == "\r") {
     
     
     if (finder.find("obTemperature") )  // this  has to be very exact.  
   {
     int temperature = finder.getValue();
    // Serial.print("Temperature is ");  //
   //  Serial.println(temperature);

   }
   else
     Serial.print("Could not find Temperature field");

     if (finder.find("obWindSpeed") )
   {
     WindSpeed = finder.getValue();
  //   Serial.print("WindSpeed is ");  //
   //  Serial.println( WindSpeed );
   }
   else
     Serial.print("Could not find WindSpeed field");
   
 }
  
 client.stop();  //Disconnect for next reload
 client.flush();//Serial.println(line);  // prints the whole page 
 Serial.println( " ");   
 Serial.print(" Temperature is ");  //
 Serial.println(Temperature);
 Serial.print(" WindSpeed is ");  //
 Serial.println( WindSpeed );
 Serial.println( " ");  
 Serial.println("Ending  Connection to EnviroCanada");
 Serial.println( " ");

--------------------------------------

But no longer works at all .
I thought this would be a matter of changing

WiFiClient client;
TextFinder finder(client);

To
WiFiClient httpsClient;
TextFinder finder(httpsClient);[/code]

and change all the other client references to httpsClient...

But it makes no difference and makes me think I’ve badly misunderstood the way Textfinder and client works

Anyone have any ideas?

Thanks David

4

Topics merged
PM sent to OP

5

If you can still get the data, then you might just to look at that and see if there are changes that cause the textfinder application to fail.

6

Many times i am using only substring function :slight_smile:
And it is working too.

7

Thanks for everyone’s quick responses. TextFinder is a library for the Arduino I found at Github through this link
https://playground.arduino.cc/Code/TextFinder/

I thought it was more widely used than it apparently is.

I am able to connect with the data from the websites, and read it into a string, but TextFinder just doesn’t seem to be getting it or reading it any more.

Is there another library people use to read and scrape strings? TextFinder was particularly useful since it could find a number value like temperature, following the word “Temperature”.

I guess I could write something like that myself, but wow that would be a lot of work.

Sorry about sending this to the wrong forum, I haven’t posted in a while.

Cheers

DmacQ