[Solved] Trying to scrape three different sources using ethernet client

Using Arduino Programming Questions
1

Hello everyone. I have once again resorted to this forum after spending hours trying to do this.

So basically what I am trying to do is scrape three different text files using the client.read(). It works perfect for a single text file but doesn't work if I want to monitor more than one text files. Here's what I mean:
There are three text files, ali1.txt, ali2.txt and ali3.txt. I want the arduino to get the values from each of these text files.
The arduino needs to grab the value from ali1.txt, and based on the value from that it needs to either scrape ali2.txt or ali3.txt

#include <SPI.h>
#include <Ethernet.h>
byte mac[] = { 
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};

EthernetClient client;
char server[] = "192.168.1.101";//Provide server to scrape data from


void setup()
{
  Serial.begin(9600);
  Ethernet.begin(mac);
  Serial.print("My IP address: ");
  Serial.println(Ethernet.localIP());
  //getdata3();
}

void loop(){
  //here I am scraping the first text file
  client.available();
  char e = client.read();
  Serial.println(e);
  client.stop();
  client.connect(server, 80);
  client.println("GET /ali1.txt");
  
  if (e==1){

    //Case 1
    char c = client.read();
    Serial.print(c);
    client.stop();
     //here I am scraping the second text file...see below for getdata1
    getdata1();
  }
  else
  {
    //Case 2
    char d = client.read();
    Serial.print(d);
    client.stop();
    getdata2();
  }

  delay(1000);
}

void getdata1() {
  client.connect(server, 80);
  client.println("GET /ali3.txt");

}

void getdata2() {
  client.connect(server, 80);
  client.println("GET /ali2.txt");

}

This is not working, and by that I mean it only properly prints the last value it scraped. I can't scrape more than one value. The other value is the ANSII character for -1 (y with two dots). Any help would be much appreciated.

2

Well, for starters, you call client.available(). Do you know why you are calling it? Perhaps you don't, otherwise you would do something with the value it returns, which is the number of bytes that have arrived after a call to UDP.parsePacket().. Oh, wait, you didn't call that. Hmm.. perhaps you should look at the documentation for the Ethernet library. It can be found at Ethernet - Arduino Reference.

3

OK well, clearly I didnt know what it did. I have been trying via trial/error method. With the code I have there, it seems to grab the value from the text files fine. I took it off and it still grabs the value right.

BUT, my question is HOW do I get value from multiple text files? Can you please help with that question?

Thanks!

4

This code should do a little of what you need.
http://playground.arduino.cc/Code/WebClient

Modify it like this, and retrieve what you need in the getPage function call.

void loop()
{
  thisMillis = millis();

  if(thisMillis - lastMillis > delayMillis)
  {
    lastMillis = thisMillis;

    strcpy(pageAdd,"/ali1.txt");
    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Ali1 fail "));
    else Serial.println(F("Ali1 pass "));

    strcpy(pageAdd,"/ali2.txt");
    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Ali2 fail "));
    else Serial.println(F("Ali2 pass "));

     strcpy(pageAdd,"/ali3.txt");
    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Ali3 fail "));
    else Serial.println(F("Ali3 pass "));
  }    
}
5

Why make the Arduino do the scraping? The server is orders of magnitude faster. Put the burden of scraping the files on the server.

6

Thanks SurferTim...I will study your code and try it out. Thanks again!

PaulS, the arduino is merely "grabbing" the value from the text files (probably used the word scraping wrong), the server is indeed doing the work of scraping/downloading, formatting it etc and putting the final value into the text file.

I just want the arduino to grab that final value. Thanks for your response too kind sir.

7

SurferTim:
This code should do a little of what you need.
Arduino Playground - HomePage

Modify it like this, and retrieve what you need in the getPage function call.

SurferTim, I get the error "getPage was not declared in this scope". Am I supposed to add anther library? I only see Ethernet and SPI in your code too. What am I missing? :frowning:

8

That was dumb...sorry, didnt notice it was a defined function.

9

Thank you sir Surfer. This worked!

:slight_smile:

10

the server is indeed doing the work of scraping/downloading, formatting it etc and putting the final value into the text file.

You should still have a script on the server that returns the contents of the named file. You'd only need to make one GET request, with three different arguments, to get the data from the file of interest.