Read text from webpage after open an URL

Creiv93 · April 15, 2021, 2:10pm

Hello everyone,
today i'm trying to read some text from webpage after open that page via URL with GET information.

Example:
1- arduino connected with the website
2- open www.link.com/page.php?info=1
3- info = 1 produce a certain print in the page: N011
4- arduino print "N011" on LCD

How can i read the "N001" in the page? i'm stuck on point 3

Thank you

J-M-L · April 15, 2021, 2:17pm

how did you send the request?

client.read() is likely part of the answer (gather all the bytes from the answer, extract what you need)

Creiv93 · April 16, 2021, 8:41am

#include <Wire.h>
#include <LiquidCrystal_I2C.h>
#include <SPI.h>
#include <stdlib.h>
#include <Ethernet.h>
#include <EthernetUdp.h>
#include <Adafruit_PN532.h>
#include <avr/wdt.h>
#include <utility/w5100.h>

#define PN532_IRQ   (2)
#define PN532_RESET (20)

Adafruit_PN532 nfc(PN532_IRQ, PN532_RESET);

#if defined(ARDUINO_ARCH_SAMD)
#define Serial SerialUSB
#endif

LiquidCrystal_I2C lcd(0x27, 20, 4);
EthernetClient client;

byte mac[] = { 0xBE, 0xEF, 0xDE, 0xAD, 0xFE, 0xED };
unsigned int localPort = 8888;
EthernetUDP Udp;
long time = 0;
long debounce = 500;
long resetdeb = 15000;
int x = 0;
// --------------------------------------------------

void(* Riavvia)(void) = 0;

// --------------------------------------------------

void setup()
{
  delay(50);
  Serial.begin(57600);
  lcd.begin();
  Wire.begin();
  SPI.begin();
  lcd.noBacklight();
  lcd.clear();
  lcd.setCursor(0, 0);
  lcd.print( F("PROVA SRL CHIAVIDISP"));
  lcd.setCursor(0, 3);
  lcd.print( F("Starting..."));
  int DHCP = 0;
  DHCP = Ethernet.begin(mac);
  lcd.setCursor(0, 3);
  lcd.print( F("Riconnessione... "));
  while ( DHCP == 0) {
    delay(1000);
    DHCP = Ethernet.begin(mac);
    lcd.print( F("RICONNESSIONE... "));
  }
  if (!DHCP) {
    Serial.println("DHCP FAILED");
  }
  else {
    Serial.println("DHCP Success");
    lcd.clear();
  }

  nfc.SAMConfig();
  wdt_enable(WDTO_2S);
  W5100.init();
  W5100.setRetransmissionTime(0x07D0);   //2000 is the default value, equivalent to 200ms
  W5100.setRetransmissionCount(2);     //8 is the default value
  client.setTimeout(500);

  Udp.begin(localPort);
}

// --------------------------------------------------

void loop()
{
  wdt_reset();

  uint8_t success;
  uint8_t uid[] = { 0, 0, 0, 0, 0, 0, 0 };
  uint8_t uidLength;

  lcd.backlight();

  lcd.setCursor(0, 1); lcd.print( F("PASSA LA TUA TESSERA"));

  success = nfc.readPassiveTargetID(PN532_MIFARE_ISO14443A, uid, &uidLength, 50);
  if (success) {
    Serial.print("UID: ");
    nfc.PrintHex(uid, uidLength);
    Serial.println();
    client.connect("prova.altervista.org", 80);
    client.print( F("GET /chiavi.php?uid="));
    for (int i = 0; i < uidLength; i++)
    {
      client.print(uid[i], HEX);
    }
    client.println( F(" HTTP/1.1"));
    client.println( F("Host: prova.altervista.org"));
    client.println( F("Connection: close"));
    client.println();

    lcd.clear();
    lcd.setCursor(0, 0);
    lcd.print("VEICOLO DESTINATO:");
    //NEED THE RESULT HERE
    client.flush();
    client.stop();
    delay(1500);
    Riavvia();
  }


  byte server[] = { 104, 28, 2, 10 };
  byte wdcount = 0;
  while (!client.connect(server, 80)) {
    lcd.clear();
    lcd.backlight();
    lcd.setCursor(0, 1);
    lcd.print( F("NESSUNA CONNESSIONE!"));
    lcd.setCursor(0, 3);
    lcd.print( F(" riprova piu' tardi "));
    Serial.println( F(" NON CONNESSO!!!"));
    delay(1000);
    if (wdcount <= 30) wdt_reset(); wdcount++;
    client.flush();
    lcd.clear();
    lcd.noBacklight();
  }
}

This is my code, it work until i need to print the result from the page "chiavi.php"

J-M-L · April 16, 2021, 9:02am

try adding

while(client.available()) {
  char c = client.read();
  Serial.print(c);
}

if(!client.connected()) {
  Serial.println("disconnected");
  client.stop();
}

in replacement of those 3 lines

//NEED THE RESULT HERE
client.flush();
client.stop();

and have a look at what gets printed to the Serial monitor (opened @57600 bauds)

Creiv93 · April 16, 2021, 9:11am

DHCP Success
TIMEOUT!
TIMEOUT!
TIMEOUT!
TIMEOUT!
TIMEOUT!
TIMEOUT!
TIMEOUT!
UID: 0xD7 0xCB 0x5B 0x52

FFFFFFFF
DHCP Success
TIMEOUT!
TIMEOUT!

This is the result on Serial Monitor
"UID: 0xD7 0xCB 0x5B 0x52" this is the RFID card uid
"FFFFFFFF" it's the print but the webpage is returning "N001"

J-M-L · April 16, 2021, 9:13am

well apparently not.... where does this FFFFFFF come from?

Creiv93 · April 16, 2021, 9:18am

I dont' really know, the webpage is very simple: when you open the page with "?uid=XXX", the page read in the database the data and print it .
So the final result is:

<html>
<head></head>
<body>N001</body>
</html>

J-M-L · April 16, 2021, 9:38am

keep things simple when you debug, try with a very crude code making the request to ensure your web server is actually responding. Try with this (typed here, so hopefully works)

#include <Ethernet.h>
EthernetClient client;

byte mac[] = { 0xBE, 0xEF, 0xDE, 0xAD, 0xFE, 0xED };

void setup() {
  while (!Serial);
  Serial.begin(57600);

  if (Ethernet.begin(mac) == 0) {
    Serial.println(F("No IP through DHCP"));
    while (true); // stop here
  }

  // make a request
  if (client.connect("prova.altervista.org", 80)) {
    Serial.println("Connected to server");
    client.print(F("GET /chiavi.php?uid=1020304050"));
    client.println(F( " HTTP/1.1"));
    client.println(F("Host: prova.altervista.org"));
    client.println(F("User-Agent: arduino-ethernet"));
    client.println(F("Connection: close"));
    client.println(); // needed to end HTTP header

    // read the answer
    while (client.connected())
      if (client.available())
        Serial.print((char) client.read());

    client.stop();
    Serial.println();
    Serial.println("disconnected");
  } else {
    Serial.println("connection failed");
  }
}

void loop() {}

what do you see in the Serial monitor ?

Creiv93 · April 16, 2021, 9:39am

Now i change my code like this:

  success = nfc.readPassiveTargetID(PN532_MIFARE_ISO14443A, uid, &uidLength, 50);
  if (success) {
    Serial.print("UID: ");
    nfc.PrintHex(uid, uidLength);
    Serial.println();
    client.connect("https://www.prova.altervista.org", 80);
    client.print( F("GET /chiavi.php?uid="));
    for (int i = 0; i < uidLength; i++)
    {
      client.print(uid[i], HEX);
    }
    client.println( F(" HTTP/1.1"));
    client.println( F("Host: https://www.prova.altervista.org"));
    client.println();

    lcd.clear();
    lcd.setCursor(0, 0);
    lcd.print("VEICOLO DESTINATO:");
    while(client.available()) {
        char c = client.read();
        Serial.print(c);
      }
      
      if(!client.connected()) {
        Serial.println("disconnected");
        client.stop();
      }
    delay(1500);
    Riavvia();
  }

and the result on Serial Monitor is


DHCP Success
UID: 0xD7 0xCB 0x5B 0x52

DHCP Success
UID: 0xD7 0xCB 0x5B 0x52

disconnected
DHCP Success
UID: 0xD7 0xCB 0x5B 0x52

HTTP/1.1 400 Bad Request
Date: Fri, 16 Apr 2021 09:35:40 GMT
Server: Apache
Content-Length: 268
Connection: close
Content-Type: text/html; charset=iso-8859-1

<html><title>Bad request</title><head></head><body><p>Anomalia nella richiesta al sito, se il problema persiste <b>cancella i cookie</b>, <a href='https://support.google.com/accounts/answer/32050?hl=it' target='_blank'>maggiori informazioni</a></li></ul></body></html>disconnected

sometime after the UID print the program jump on Riavvia() -> reaload
sometimes print disconnected
and sometimes print bad request (the URL point at the right page)

J-M-L · April 16, 2021, 9:40am

try the code I posted above.

Creiv93 · April 16, 2021, 9:58am

Connected to server
HTTP/1.1 301 Moved Permanently
Date: Fri, 16 Apr 2021 09:57:38 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Set-Cookie: __cfduid=d4c965c1d91a00aeddf3d5143921d596d1618567058; expires=Sun, 16-May-21 09:57:38 GMT; path=/; domain=.prova.altervista.org; HttpOnly; SameSite=Lax
Location: https://prova.altervista.org/chiavi.php?uid=1020304050
CF-Cache-Status: DYNAMIC
cf-request-id: 097bb66b4c00004c975a9ea000000001
Report-To: {"max_age":604800,"group":"cf-nel","endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=5q4IYORnzFbX7A6mr79gmXMiDUPm%2FqbxR5OHq24o5mqZUbfbdRY6KiZGpWRHyx0PGAIMcv2YqPhZmfx%2B3ajJql%2Bv6%2BOUB8esCLYQZGKcsnFC%2Fgp0yg%3D%3D"}]}
NEL: {"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 640c8cf1dec44c97-AMS
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

0


disconnected

this is the result

J-M-L · April 16, 2021, 10:08am

so the good news is that you do get an answer.
The bad news is that it does not look like this at all

<html>
<head></head>
<body>N001</body>
</html>

I'm sending a GET for /chiavi.php?uid=1020304050 --> would an incorrect UID generate the response you saw? can you try with a legit UID instead of 1020304050?

Creiv93 · April 16, 2021, 11:26am

I modified my webpage like this:

<?php
	 require "requisitix.php";

	 $connessione=mysql_connect($ip, $username, $password) or die("Connessione fallita. " . mysql_error());

	 mysql_select_db(my_prova, $connessione);

	 $uid= $_GET['uid'];
	 date_default_timezone_set("Europe/Rome");
	 $data = date("Y-m-d");
	 $ora = date("H:i");

	 $query = "SELECT * FROM turnimezzi WHERE data = '$data' AND idutente = '$uid' ORDER BY id DESC";
	 $result = mysql_query($query, $connessione);
	 $record = mysql_fetch_array($result);
   //echo $record['mezzo'];
echo "N001";

?>

So my page now have only to print "N001"

J-M-L · April 16, 2021, 12:30pm

can you try with

<?php
echo "N001";
?>

just to make sure nothing else gets in the way ?

Creiv93 · April 16, 2021, 1:01pm

Connected to server
HTTP/1.1 400 Bad Request
Date: Fri, 16 Apr 2021 13:01:27 GMT
Server: Apache
Content-Length: 268
Connection: close
Content-Type: text/html; charset=iso-8859-1

<html><title>Bad request</title><head></head><body><p>Anomalia nella richiesta al sito, se il problema persiste <b>cancella i cookie</b>, <a href='https://support.google.com/accounts/answer/32050?hl=it' target='_blank'>maggiori informazioni</a></li></ul></body></html>
disconnected

this is the result

J-M-L · April 16, 2021, 1:27pm

OK
So you have some issues with the request

HTTP/1.1 400 Bad Request

the web server is not forwarding it to the PHP script

May be have a look at

Creiv93 · April 16, 2021, 1:39pm

Oh, maybe the problem is because of Cloudflare

Creiv93 · April 16, 2021, 2:11pm

I found the problem with your sketch:

 client.print(F("GET "));
 client.print("/chiavi.php");  <------ the page into the print without the F
 client.print(F("?uid=010101"));

and

 client.println(F("Host: prova.altervista.org")); <------ without any \ or /

and at last but not the least

(client.connect("https://www.prova.altervista.org", 80)) <-------- with "https://www."

Hope to help someone, have a good day and thank you!

J-M-L · April 16, 2021, 2:40pm

this one should not be an issue at all. the bytes are sent in the same way

I had missed the https, this is unsupported wit the ethernet card !

system · August 14, 2021, 2:41pm

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Reading Txt String From WebPage Programming	5	1114	May 5, 2021
Arduino Ethernet reading webserver input Programming	6	1554	May 5, 2021
Arduino Ethernet reading XML Programming	3	1000	May 5, 2021
[Solved] how i can read a php page? Networking, Protocols, and Devices	3	619	May 6, 2021
Viewing text from website on display General Guidance	4	697	May 5, 2021

Read text from webpage after open an URL

Related topics