I have an equipment that I connect to on LAN with my laptop and change some of its parameters from time to time. It has a webpage which I access and change the parameters that I need.
I have an arduino with w5500 elsewhere in the system and connected to a switch common with the equipment mentioned above.
I need to automate configuration of a particular parameter based on certain conditions. I want to do it with arduino.
Now the problem is that, the particular page where the parameter of interest exists has a GET-method HTML form.( please see the attachment) I plan to mimic the action performed when submit button is clicked on the webpage with arduino. I want to send the same GET method data so that the equipment will think that the command is coming from the webpage.
When I click the button all the data of the form goes to the server in the equipment, along with the change that I made manually. But when I want to do the same thing with arduino if I send only the variable that I want to change, the equipment discards the data as probably it gets all other variables as zero by default if I do not send them.
So, what I understand is that I need to pull up the page, parse the HTML, check all variables for their values and send all of them along with the change that I want to make.
But it is around 2K bytes of data which is too much for a Uno to handle. I can go to a mega but I still think it is highly inefficient.
I just want some ideas on smarter ways to do what I am trying to do.
(I cannot change the equipment side of the codes)
I hope my problem statement is not too confusing.
If that works, you should then generate the same string with an Arduino. The format is "Query String".
You could also look at the source HTML of that web page by using a browser function to see how it is built up.
You talk about 2k ? Does that include all the text, images, formatting or just the pure data ? If you want to interpret a web page created by another system, you may have difficulty parsing the desired data out from all that.
Thanks for the reply.
My first instinct was also to do what you told, send only the relevant data in the query string. But unless I make the query string to contain data of all the input fields the equipment does not update the values. Maybe they have a check on their side.
When I say this page is 2k I mean that when I see the HTML data that I receive when I access this page it has around 1900 characters. So, one solution may be to query the webpage, store the data, parse the data, get all values and make a query to send. But, I want to find out if their is a simpler and smarter way to do it
This is the code on my arduino. You can see the query that I have hardcoded for testing
#include <SPI.h>
#include <Ethernet2.h>
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(192, 168, 222, 221);
EthernetClient client;
void setup() {
Serial.begin(115200);
Ethernet.begin(mac, ip);
delay(1000);
Serial.println("connecting...");
while (!client.connect("192.168.222.222", 80));
Serial.println("connected");
}
void loop()
{
char voltage_level = 0;
if (!client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
}
Serial.println("For 13V press 1\nFor 18V press 2");
while (!Serial.available());
if (Serial.available())
voltage_level = Serial.read();
if (voltage_level == '1')
{
client.println("GET /cR3?dn=0&dj=0&da=5&db=1210000&dc=32610&dm=0&dk=0&dh=952000&di=1000&do=0&dl=1&tf=Apply HTTP/1.1"); // THIS WORKS
// client.println("GET /cR3?dl=1&tf=Apply HTTP/1.1"); //THIS DOES NOT WORK
client.println("Host: 192.168.222.222");
client.println("Connection: close");
client.println();
}
else if (voltage_level == '2')
{
client.println("GET /cR3?dn=0&dj=0&da=5&db=1210000&dc=32610&dm=0&dk=0&dh=952000&di=1000&do=0&dl=0&tf=Apply HTTP/1.1");
// client.println("GET /cR3?dl=0&tf=Apply HTTP/1.1");
client.println("Host: 192.168.222.222");
client.println("Connection: close");
client.println();
}
}
Filling forms programmatically is not always straightforward. Mainly because:
1- There are almost always cookies involved. These cookies are acquired through the steps it takes you to reach the form. For example you get some cookies after you log in, some other cookies when you open the form page, etc. If you want to replicate this behavior in your code you have to acquire those cookies as well.
2- There might be JavaScript code on the page that calculates certain values and sends them to the server. If you want to submit the form from your own code you will have to study that JavaScript code and find out how it calculates those values and implement that in your own code.
So what are you gonna do?
First you need to figure out exactly what request is being made. Make sure it's a GET request. Find out all the parameters. See if there are any cookies. See if there are certain HTTP headers sent. etc.
For that you can use a web debugging software such as Fiddler, or your browser's web development tools.
Every major browser has web development tools that has a "network" tab that will allow you to inspect all the HTTP requests your browser is making. Upon clicking the "submit" button a request will be made. Study that request and see what parameters and cookies are being sent.
After that you want to replicate the request using cURL. It's a tool for making HTTP requests and is available for both Windows and Linux. The reason we use cURL is because it's something inbetween a browser and an Arduino. It's more "raw" so to speak.
If you can manage to achieve what you want using cURL then you will be able to do the same with an Arduino. cURL has the advantage that it's on your PC and you can quickly change parameters until you have achieved your goal.
If you use Firefox you can right click on a request in the 'network' tab of developer tools and click copy->copy as curl. The command that it gives you has all the information you have to send to the server in order to replicate that request.
But again, some of the parameters might be mysterious numbers that you have to figure out.
Or maybe it will be straightforward. You have to try to find out.
This is how I would do it. It might have a bit of a learning curve but what you're trying to do isn't as simple as it might sound for the reasons I explained.
pourduino:
First you need to figure out exactly what request is being made. Make sure it's a GET request. Find out all the parameters. See if there are any cookies. See if there are certain HTTP headers sent. etc.
Hi,
Thanks for your inputs.
There is no cookies and scripts other than plain HTML. It is a simple maintenance GUI of an equipment without any login requirements. And it is a GET request as I have verified with the source code of the page and also with wireshark checking the data that is sent when I click the submit button.
Also, when I send the same query which I got from wireshark and modify according to what I want and send it from arduino it works.
Problem arises only when I want to change only one of the input fields as I have mentioned in the code I have attached.
But again, some of the parameters might be mysterious numbers that you have to figure out.
I have figured out all the parameters that need to be sent. The problem is changing only one of them without disturbing others.
OK. You have got quite far. I certainly wasn't suggesting that you attempt to send only a subset of the data. Clear is that it has to be complete.
I guess that the data on that web page is neither static nor easy to determine without fetching the web page. That is you are forced to retrieve the web page to collect the data before you can prepare the query string to send.
Do you have access to the device which is generating that web page to be able to create your own, more simply formatted web page, say as json or XML ? Otherwise, you have no option but to parse out the required data from the existing web page, edit it as required, then send it back to the server.
If you show the source of that HTML web page, someone may be able to suggest a simple and reliable way of parsing it, however, the success of this depends on how "fixed" the format of that web page is.
The webpage is not really under my control. But it is pretty fixed.
Below is the code from the webpage. I have changed the names of a few parameters. I feel parsing may be the only option. I need to change the value of the last radio button only from arduino.
Now I need to figure out whether I can parse it part by part in small chunks so that I don't need to save whole 2K bytes in memory at the same time.
You might be able to read it line by line, and store only the lines that matter.
For example you need line 39, 50, 55, and so on.
That's assuming the actual HTML has new lines in it (\r\n) and that the structure of the HTML it sends is always the same.
pourduino:
You might be able to read it line by line, and store only the lines that matter.
For example you need line 39, 50, 55, and so on.
That's assuming the actual HTML has new lines in it (\r\n) and that the structure of the HTML it sends is always the same.
I will try this. Thanks. The structure of the page is static. It doesn't change at all.
The source code doesn't seem to have any newline characters when I get it. But I will try saving only when the Form entries start with "INPUT name=" as anchor points.
You can certainly do it with C++, but it is not going to be easy. Ideal would be a language which supports regular expressions and maybe a hash table for the intermediate results. I guess I'd do this by getting another, more capable platform (say a WAMP server) in your network to fetch the page, do the parsing and present the results simply as JSON like say an API for a weather service or similar does.
Thanks for the suggestions.As you suggested, I guess it is better to move this to a more powerful platform if possible. I will explore this possibility.
