Pages: [1]   Go Down
Author Topic: [Solved] Trying to scrape three different sources using ethernet client  (Read 351 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 17
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hello everyone. I have once again resorted to this forum after spending hours trying to do this.

So basically what I am trying to do is scrape three different text files using the client.read(). It works perfect for a single text file but doesn't work if I want to monitor more than one text files. Here's what I mean:
There are three text files, ali1.txt, ali2.txt and ali3.txt. I want the arduino to get the values from each of these text files.
The arduino needs to grab the value from ali1.txt, and based on the value from that it needs to either scrape ali2.txt or ali3.txt

Code:
#include <SPI.h>
#include <Ethernet.h>
byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};

EthernetClient client;
char server[] = "192.168.1.101";//Provide server to scrape data from


void setup()
{
  Serial.begin(9600);
  Ethernet.begin(mac);
  Serial.print("My IP address: ");
  Serial.println(Ethernet.localIP());
  //getdata3();
}

void loop(){
  //here I am scraping the first text file
  client.available();
  char e = client.read();
  Serial.println(e);
  client.stop();
  client.connect(server, 80);
  client.println("GET /ali1.txt");
  
  if (e==1){

    //Case 1
    char c = client.read();
    Serial.print(c);
    client.stop();
     //here I am scraping the second text file...see below for getdata1
    getdata1();
  }
  else
  {
    //Case 2
    char d = client.read();
    Serial.print(d);
    client.stop();
    getdata2();
  }

  delay(1000);
}

void getdata1() {
  client.connect(server, 80);
  client.println("GET /ali3.txt");

}

void getdata2() {
  client.connect(server, 80);
  client.println("GET /ali2.txt");

}

This is not working, and by that I mean it only properly prints the last value it scraped. I can't scrape more than one value. The other value is the ANSII character for -1 (y with two dots). Any help would be much appreciated.
« Last Edit: February 16, 2014, 03:33:52 pm by mfarooqalikhan » Logged

Saskatchewan, Canada
Offline Offline
Edison Member
*
Karma: 49
Posts: 1419
Coding Geezer
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well, for starters, you call client.available(). Do you know why you are calling it?  Perhaps you don't, otherwise you would do something with the value it returns, which is the number of bytes that have arrived after a call to  UDP.parsePacket().. Oh, wait, you didn't call that. Hmm.. perhaps you should look at the documentation for the Ethernet library. It can be found at http://arduino.cc/en/Reference/Ethernet.
Logged

There are 10 kinds of people in the world,
those who understand binary, and those who don't.

Offline Offline
Newbie
*
Karma: 0
Posts: 17
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

OK well, clearly I didnt know what it did. I have been trying via trial/error method. With the code I have there, it seems to grab the value from the text files fine. I took it off and it still grabs the value right.

BUT, my question is HOW do I get value from multiple text files? Can you please help with that question?

Thanks!

Logged

Miramar Beach, Florida
Offline Offline
Faraday Member
**
Karma: 148
Posts: 6105
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This code should do a little of what you need.
http://playground.arduino.cc/Code/WebClient

Modify it like this, and retrieve what you need in the getPage function call.
Code:
void loop()
{
  thisMillis = millis();

  if(thisMillis - lastMillis > delayMillis)
  {
    lastMillis = thisMillis;

    strcpy(pageAdd,"/ali1.txt");
    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Ali1 fail "));
    else Serial.println(F("Ali1 pass "));

    strcpy(pageAdd,"/ali2.txt");
    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Ali2 fail "));
    else Serial.println(F("Ali2 pass "));

     strcpy(pageAdd,"/ali3.txt");
    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Ali3 fail "));
    else Serial.println(F("Ali3 pass "));
  }   
}
Logged

Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 633
Posts: 50208
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Why make the Arduino do the scraping? The server is orders of magnitude faster. Put the burden of scraping the files on the server.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 17
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks SurferTim...I will study your code and try it out. Thanks again!

PaulS, the arduino is merely "grabbing" the value from the text files (probably used the word scraping wrong), the server is indeed doing the work of scraping/downloading, formatting it etc and putting the final value into the text file.

I just want the arduino to grab that final value. Thanks for your response too kind sir.

Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 17
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This code should do a little of what you need.
http://playground.arduino.cc/Code/WebClient

Modify it like this, and retrieve what you need in the getPage function call.


SurferTim, I get the error "getPage was not declared in this scope". Am I supposed to add anther library? I only see Ethernet and SPI in your code too. What am I missing?  smiley-sad
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 17
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

That was dumb...sorry, didnt notice it was a defined function.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 17
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thank you sir Surfer. This worked!

 smiley
Logged

Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 633
Posts: 50208
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
the server is indeed doing the work of scraping/downloading, formatting it etc and putting the final value into the text file.

You should still have a script on the server that returns the contents of the named file. You'd only need to make one GET request, with three different arguments, to get the data from the file of interest.
Logged

Pages: [1]   Go Up
Jump to: