Hi everybody,
I'm trying to make a weather station that indicates the weather based on some datas collected on the internet (weather forecast websites). I know I could web scrape using python and letting python and arduino communicate with eachother, but I'd like to know if there's any way to web scrape direcly using arduino.
Thank you in advance.
Yes, it's possible. But more difficult. What model of Arduino do you want to use? Why do you want to use an Arduino?
A simpler solution might to be use a microcontroller board that can run python itself. For example a rp2040 based board. Of course, if you run python on a board, it can't be called an "Arduino". But it can do similar things to an Arduino such as lighting LEDs or activating motors etc.
I'm using an Arduino because it's the only one I have and I'd like to spend the least money possible . My idea is to take weather data such as "cloudy", "sunny", "rainy" and moving a servomotor on a 3d printed table so that it indicates the right image of the weather (i.e. if the website says cloudy the servo will indicate the image of a cloud). I need something to read into the HTML code of the website and that could also move a servomotor and turn on some LEDs (all this without being wired to the computer). Is there a board you would suggest me? Thanks a lot
There are several boards you could use. But you seem to want to use the board you have. I asked a question about that but you didn't answer.
Do you have a weather website in mind? Can you post a link?
"Web scraping" is a pretty broad subject which is why PaulRB's request for a sample website is relevant. Some websites are pretty large which has memory implications for the processor doing the scraping before one even gets to the mechanics of extracting information from a website.
On the other hand, some web services have APIs (e.g. JSON) for getting specific information from the internet which is considerably more compact and ultimately less complicated to implement.
I'd suggest you've posed an XY Problem, that is, a problem that results from a proposed solution to what you want to achieve, rather than describing directly what you want to achieve. There may well be a better route to your ultimate goal that does not involve "web scraping" per se.
Oh sorry, I forgot to. I have a website but it's in Italian, I don't know if you can understand it. It's "meteo.it". If you have a better option to suggest, I'm open to anything.
Thank you Mark. Do you suggest any other particular way to approach my problem?
I'm in the US and our National Weather Service has a documented API for accessing various data from NWS stations. I would be surprised if there weren't something similar for the stations you are interested in.
With an ESP32 and ArduinoJson.org library it's fairly easy to query the OpenWeatherMap.org API:
#include <WiFi.h>
#include <WiFiMulti.h>
#include <HTTPClient.h>
#include <MyHomeWiFi.h>
WiFiMulti WiFiMulti;
#include <ArduinoJson.h>
// set location and API key
String Location = "Maynard, MA, USA";
String API_Key = "eb9274010b722f75502bbf8fa15f4872";
void setup()
{
Serial.begin(115200);
delay(200);
WiFiMulti.addAP(MyHomeWiFiSSID, MyHomeWiFiPASSWORD);
Serial.print("Connecting.");
}
void loop()
{
// wait for WiFi connection
if ((WiFiMulti.run() == WL_CONNECTED))
{
WiFiClient client;
HTTPClient http;
String payload;
// specify request destination
Serial.print("[HTTP] begin...\n");
if (http.begin(client, "http://api.openweathermap.org/data/2.5/weather?q=" + Location + "&APPID=" + API_Key)) // HTTP
{
Serial.print("[HTTP] GET...\n");
// start connection and send HTTP header
int httpCode = http.GET();
// httpCode will be negative on error
if (httpCode > 0) // check the returning code
{
// HTTP header has been send and Server response header has been handled
Serial.printf("[HTTP] GET... code: %d\n", httpCode);
// file found at server
if (httpCode == HTTP_CODE_OK || httpCode == HTTP_CODE_MOVED_PERMANENTLY)
{
payload = http.getString();
Serial.println(payload);
}
}
else
{
Serial.printf("[HTTP] GET... failed, error: %s\n",
http.errorToString(httpCode).c_str());
return;
}
/*
{"coord":{"lon":77.75,"lat":20.9333},
"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],
"base":"stations","main":{"temp":301.45,"feels_like":303.3,"temp_min":301.45,
"temp_max":301.45,"pressure":1004,"humidity":62,"sea_level":1004,"grnd_level":966},
"visibility":10000,"wind":{"speed":3.14,"deg":293,"gust":7.39},"clouds":{"all":94},
"dt":1628958953,"sys":{"country":"IN","sunrise":1628900858,"sunset":1628947200},
"timezone":19800,"id":1278718,"name":"AmrÄvati","cod":200}
*/
StaticJsonDocument<1024> doc;
DeserializationError error = deserializeJson(doc, payload);
if (error)
{
Serial.print(F("deserializeJson() failed: "));
Serial.println(error.f_str());
return;
}
// float coord_lon = doc["coord"]["lon"]; // 77.75
// float coord_lat = doc["coord"]["lat"]; // 20.9333
//
// JsonObject weather_0 = doc["weather"][0];
// int weather_0_id = weather_0["id"]; // 804
// const char* weather_0_main = weather_0["main"]; // "Clouds"
// const char* weather_0_description = weather_0["description"]; // "overcast clouds"
// const char* weather_0_icon = weather_0["icon"]; // "04n"
//
// const char* base = doc["base"]; // "stations"
//
JsonObject main = doc["main"];
float main_temp = main["temp"]; // 301.45
// float main_feels_like = main["feels_like"]; // 303.3
// float main_temp_min = main["temp_min"]; // 301.45
// float main_temp_max = main["temp_max"]; // 301.45
float main_pressure = main["pressure"]; // 1004
int main_humidity = main["humidity"]; // 62
// int main_sea_level = main["sea_level"]; // 1004
// int main_grnd_level = main["grnd_level"]; // 966
//
// int visibility = doc["visibility"]; // 10000
//
JsonObject wind = doc["wind"];
float wind_speed = wind["speed"]; // 3.14
int wind_deg = wind["deg"]; // 293
// float wind_gust = wind["gust"]; // 7.39
//
// int clouds_all = doc["clouds"]["all"]; // 94
//
// long dt = doc["dt"]; // 1628958953
//
// JsonObject sys = doc["sys"];
// const char* sys_country = sys["country"]; // "IN"
// long sys_sunrise = sys["sunrise"]; // 1628900858
// long sys_sunset = sys["sunset"]; // 1628947200
//
// int timezone = doc["timezone"]; // 19800
// long id = doc["id"]; // 1278718
// const char* name = doc["name"]; // "AmrÄvati"
// int cod = doc["cod"]; // 200
// print data
Serial.printf("Temperature = %.2fĀ°C\r\n", main_temp);
Serial.printf("Humidity = %d %%\r\n", main_humidity);
Serial.printf("Pressure = %.3f bar\r\n", main_pressure);
Serial.printf("Wind speed = %.1f m/s\r\n", wind_speed);
Serial.printf("Wind degree = %dĀ°\r\n\r\n", wind_deg);
}
else
{
Serial.println("[HTTP] .begin() failed");
}
http.end(); //Close connection
}
delay(60000); // wait 1 minute
}
// End of code.
Wow this is impressive, thank you so much John!
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.