Arduino client to read data from a page running JavaScript?

Hello!
For a while now I have been wanting/trying to simply get a number from a java script page and display it on a LCD.
While printing data to an LCD and accessing an HTML page is easy, is it even possible to grab data from a page that has JavaScript controlling a number that I want?

I have seen tutorials that get simple data like this one from bildr.org but I don’t see how to make it work for what I am trying to do.

Basically, I want to read the percentage in the middle of the page from this (fictitious zip code)and print it to the serial monitor for now.

I used Firefox “inspect element” (see attachment) and found the “1%” chance of a snow day.

Is this a possibility? If so, where do I start?

Any help would be greatly appreciated!

You need to edit your OriginalPost and change "Java" to "Javascript". They are two completely different programming languages.

An Arduino is completely incapable of running a Javascript interpreter.

I guess it may be possible on the Linux side of a Yun. In the past I wrote a Python (or was it Ruby) program to screen-scape a website with Javascript. It was quite complex and required a library that acted like a browser - sorry I can't recall the details.

...R

Thanks for the reply.

I guess I will find a simpler website that displays the same data in HTML or PHP.

Or...one of my 3 Raspberry Pi's may be more suited for the task but may take a lot of time to make it work.

Thanks for the clarification in any case!

RPi sounds like a good option.

…R

Some code that gets data from a weather web page.

//////////////////////////////////////////////
// Get XML formatted data from the web.
// 1/6/08 Bob S. - Created
//  Assumptions: single XML line looks like: 
//    <tag>data</tag> or <tag>data 
// Include description files for other libraries used (if any)
//
// updated at later dates zoomkat
//////////////////////////////////////////////

#include <SPI.h>
#include <Ethernet.h>

// Define Constants
// Max string length may have to be adjusted depending on data to be extracted
#define MAX_STRING_LEN  20

// Setup vars
char tagStr[MAX_STRING_LEN] = "";
char dataStr[MAX_STRING_LEN] = "";
char tmpStr[MAX_STRING_LEN] = "";
char endTag[3] = {'<', '/', '\0'};
int len;

// Flags to differentiate XML tags from document elements (ie. data)
boolean tagFlag = false;
boolean dataFlag = false;

// Ethernet vars
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
//byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 113, 200 }; // www.weather.gov

// Start ethernet client
EthernetClient client;

void setup()
{
  Serial.begin(9600);
  Serial.println("Starting WebWx");
  Serial.println("connecting...");
  //Ethernet.begin(mac, ip);
  Ethernet.begin(mac);
  delay(1000);

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");    
    client.println();
    delay(2000);
  } else {
    Serial.println("connection failed");
  }  
}

void loop() {

  // Read serial data in from web:
  while (client.available()) {
    serialEvent();
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("Disconnected");
    Serial.println("==================================");
    Serial.println("");
    client.stop();

    // Time until next update
    //Serial.println("Waiting");
    for (int t = 1; t <= 15; t++) {
      delay(60000); // 1 minute
    }

    if (client.connect(server, 80)) {
      //Serial.println("Reconnected");
      client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");    
      client.println();
      delay(2000);
    } else {
      Serial.println("Reconnect failed");
    }      
  }
}

// Process each char from web
void serialEvent() {

   // Read a char
	 char inChar = client.read();
   //Serial.print(".");
  
   if (inChar == '<') {
      addChar(inChar, tmpStr);
      tagFlag = true;
      dataFlag = false;

   } else if (inChar == '>') {
      addChar(inChar, tmpStr);

      if (tagFlag) {      
         strncpy(tagStr, tmpStr, strlen(tmpStr)+1);
      }

      // Clear tmp
      clearStr(tmpStr);

      tagFlag = false;
      dataFlag = true;      
      
   } else if (inChar != 10) {
      if (tagFlag) {
         // Add tag char to string
         addChar(inChar, tmpStr);

         // Check for </XML> end tag, ignore it
         if ( tagFlag && strcmp(tmpStr, endTag) == 0 ) {
            clearStr(tmpStr);
            tagFlag = false;
            dataFlag = false;
         }
      }
      
      if (dataFlag) {
         // Add data char to string
         addChar(inChar, dataStr);
      }
   }  
  
   // If a LF, process the line
   if (inChar == 10 ) {

/*
      Serial.print("tagStr: ");
      Serial.println(tagStr);
      Serial.print("dataStr: ");
      Serial.println(dataStr);
*/

      // Find specific tags and print data
      if (matchTag("<temp_f>")) {
	      Serial.print("Temp: ");
         Serial.print(dataStr);
      }
      if (matchTag("<relative_humidity>")) {
	      Serial.print(", Humidity: ");
         Serial.print(dataStr);
      }
      if (matchTag("<pressure_in>")) {
	      Serial.print(", Pressure: ");
         Serial.print(dataStr);
         Serial.println("");
      }

      // Clear all strings
      clearStr(tmpStr);
      clearStr(tagStr);
      clearStr(dataStr);

      // Clear Flags
      tagFlag = false;
      dataFlag = false;
   }
}

/////////////////////
// Other Functions //
/////////////////////

// Function to clear a string
void clearStr (char* str) {
   int len = strlen(str);
   for (int c = 0; c < len; c++) {
      str[c] = 0;
   }
}

//Function to add a char to a string and check its length
void addChar (char ch, char* str) {
   char *tagMsg  = "<TRUNCATED_TAG>";
   char *dataMsg = "-TRUNCATED_DATA-";

   // Check the max size of the string to make sure it doesn't grow too
   // big.  If string is beyond MAX_STRING_LEN assume it is unimportant
   // and replace it with a warning message.
   if (strlen(str) > MAX_STRING_LEN - 2) {
      if (tagFlag) {
         clearStr(tagStr);
         strcpy(tagStr,tagMsg);
      }
      if (dataFlag) {
         clearStr(dataStr);
         strcpy(dataStr,dataMsg);
      }

      // Clear the temp buffer and flags to stop current processing
      clearStr(tmpStr);
      tagFlag = false;
      dataFlag = false;

   } else {
      // Add char to string
      str[strlen(str)] = ch;
   }
}

// Function to check the current tag for a specific string
boolean matchTag (char* searchTag) {
   if ( strcmp(tagStr, searchTag) == 0 ) {
      return true;
   } else {
      return false;
   }
}

zoomkat: Some code that gets data from a weather web page.

//////////////////////////////////////////////
// Get XML formatted data from the web.

But the data is not generated by Javascript !

...R

But the data is not generated by Javascript !

I would suspect that the same general methods of capturing data out of a data stream would be similar no mater what is generating the stream. One would just have to adapt to the way the data stream is formatted.

zoomkat: I would suspect that the same general methods of capturing data out of a data stream would be similar no mater what is generating the stream. One would just have to adapt to the way the data stream is formatted.

I suspect not. I tried to capture data from the UK Met office web page and it was essential to process the page with a Javascript interpreter to produce the data before I could capture it. The initial web page had javascript in it but did not contain the data. The javascript was used to get the data.

...R

True, if the javascript is running in the browser to display in the browser, then the arduino client is out of the picture.