Pull down info from webpage using XPATH or CSS selector? ESP8266 as Arduino

Projects Programming
1

Hi all,
I'm trying to use an esp8266 programmed as an Arduino to go out and pull down a specific section of a webpage and print the plain text (numerical) to the serial monitor.

Thanks in advance,

triops.

2

How, exactly, are you trying that? What problem(s) are you having?

3

I don't know where to start. I wasn't able to find any code examples online that use esp8266 using Arduino IDE to query information from a specific portion of a website. Should it use HTTP GET or HTTP POST? how? I need to see a basic code skeleton and work from there.

4

I need to see a basic code skeleton and work from there.

What you NEED is to work out, on a PC, how to achieve the results you are looking for. Then, you can implement the process on the Arduino/ESP.

5

Im looking for a similar method with just the esp8266 and no third party services. In the mean time I have had a lot of luck using ThingSpeak's ThingHTTP service. It is capable of parsing xpath (including from HTTPS sites, which is nice because I haven't been able to figure out how to use the HTTPS libraries properly apparently) and sending that parsed string to your esp. There is an access limit of one request per 15 seconds for the free version. I will post here if I ever figure out a non third-party method.
For more info on ThingHTTP and esp8266, just google it, there are a number of good tutorials and examples out there already.

6

If the XML file isn't too complex, you don't actually need to parse the entire file. If you're just looking for a specific tag, it's actually really easy.

Here's an example sketch I wrote to get the number of unread emails from a Gmail XML file. It also demonstrates the use of HTTPS.

https://tttapa.github.io/ESP8266/Chap17%20-%20Email%20Notifier.html

  int unread = -1;

  while (client.connected()) {                          // Wait for the response. The response is in XML format
    client.readStringUntil('<');                        // read until the first XML tag
    String tagname = client.readStringUntil('>');       // read until the end of this tag to get the tag name
    if (tagname == "fullcount") {                       // if the tag is <fullcount>, the next string will be the number of unread emails
      String unreadStr = client.readStringUntil('<');   // read until the closing tag (</fullcount>)
      unread = unreadStr.toInt();                       // convert from String to int
      break;                                            // stop reading
    }                                                   // if the tag is not <fullcount>, repeat and read the next tag
  }

Pieter

7

A fully-fledged XPATH or CSS parser is probably not going to fit on an arduino.

An XML parser using a streaming API might.

8

Thank you all for the help. An adaptation of PieterP's answer looks like it will work perfectly for my purposes. Very helpful!

While I was waiting for these answers, I got a bit sidetracked and taught myself how to write a couple python command line apps for doing something like this on a PC. It could easily be adapted to ring a bell on a raspberry pi using a servo or something.

This one can work with stats from multiple sites:GitHub - donovanmagryta/Stats-Bell: Python CLI app for monitoring social media stats

And this second one has text-to-speech like Siri or Alexa and specifically monitors views on instructables.com:https://www.instructables.com/id/Instructable-Hit-Announcer-Bot/

Sorry for the tangent, but it was interesting to me, and hopefully to y'all too. Have a nice evening!