getting data from a website

Hi,
I have bought esp8266 module for my project.
I Have already experience with html, javascript and the esp8266 wifi libraries.
My ultimate goal is to be able to access some data on a website, and use it in the arduino. Specifically, a have a somewhat complicated program(in Javascript) that produces variables (up to 1000 variables, organized in an array in the javascript sketch, just FIY). So the esp8266 needs in some way to be able to read theese variables and storing them(so that i can use them to control a couple of steppers).
with the esp-8266 i have successfully been able to obtain strings from the internet with http get requests on this website(http://jsonplaceholder.typicode.com/users), but nothing else. i tried this way (http get requests) with a simple html page with just some text, but it doesnt work. I think i just dont understand enough http. so i am asking if there is a more simple way to do it.

What you are trying to do is quite normal.
I do it myself.

Two thing to check
Is the request_string longer than 200 (or 500).
I think I had an issue once sending long /GET arguments.
But receiving should be fine and only limited by ram.

Oh 2nd'ly are you getting any response.?
Use the serial print to show you any data and at the point it is failing.

The easiest method is to actually save the dataset to a file on the server and then just download it to the SPIFFS. From there you can work on it at leisure.
Pretty much the same as just downloading the data raw.

If you need some example code I can paste some but all I know I got from libraries and examples

I dont know how or why, now i do actually get a response.
before i was printing everything in the serial monitor and i could see the status code of the get request was -1 (so the get request was outright failing). now, with the same code on the arduino i can print the whole html page. The test page i have created and uploaded is http://hamster-194.getforge.io/ , and the arduino now prints (as it should):

200

this is my website

x++
0

the total data that i will need is probably going to be bigger than 500 characters, but i see no problem in sending it in pieces, using the string(of max length wichever it might be, since you say there is a max), deleting it, and then send in the next piece. (that is an option if necessary)

"The easiest method is to actually save the dataset to a file on the server and then just download it to the SPIFFS. From there you can work on it at leisure.
Pretty much the same as just downloading the data raw."

you mean, in the javascript save all the variables in a separate file (json format i assume?,idk another), and then do the get request on that file (so on the url www.mywebsite.com/file.json) so that then a have a string on the esp8266 with all the variables. and i can run through all the characters in the string, and get the useful data out of it?
if so how exactly would i save all the variables of the javascript in a separate file? like, have already setup a blank file, and then with some function file.write(variables)?

i quickly looked about SPIFFS here (A Beginner's Guide to the ESP8266), but if you could summarise it would be very helpful(otherwise no worry ill spend more time on it).

Thank You.

FIY, i will use the p5.js web editor to create the javascript, and so have the script in a separte sketch.js file in the website folder.

FIY, if i try to do a get request on the same test website above, with another program (a quick index.html file that i wrote on Atom, that i open in the browser, i get the CORS policy missing ACAO header error, but since from the esp8266 it seems to work, for now its not an issue).

Ahh,
Coding is like that, one day it works, next day it want's to rest.
there is always a reason but finding it is half the fun.

Anyway, Forget SPIFFS for right now,
It is simple enough but it just adds another learning layer before it is needed.
If you need storage for up to 1k then simulated EEPROM is also fine.

All the saving as a file first talk was me just complicating things, As you can see, in reality you get to see the FULL contents of what was echo'd back from the webserver, plus a little header.
So it does not matter what you request it all just comes back as data to be read.

So in your "PHP"?? you can just "echo" back any response you want and catch it in the ESP as you are currently doing. If you do not have access to PHP on the server and are stuck with the presented format then you just have to parse the response you get and extract the required parts, that may require you downloading secondary files that actually contain the data you need.

JavaScript is more a client side script to run on the browser end, so not that handy to the ESP as it needs reading and parsing. Thus all the SessionStorage tools is also not available.

Receiving 1k of data is fine, you can receive as much as you have RAM,
Transmitting large chunks out of the ESP is also fine but there is a limit on how much stuff you can put in the URL as arguments so you may have to "/POST" the data.

for the future, If you can run "websockets" on your server then you could have the ESP maintain a connection with the server and then the server could also push the data without the need for the ESP to request it, but it's another learning layer again thus for the future.

I do not know enough about the getforge.io tools to know what you can and cannot do but I suspect they will have methods to push and pull the data you need without the complications.

getforge.io is just a fast and free hosting solution that i found, to test if what i'm trying to do works before entering billing information on another more "official" web hosting site. And i'm not sure either if the servers have access to php (but don't you just include in your website folder the php file, and do what you gotta do?...i'm not a php expert).

Currently the only option i see is what i suggested in my previous post (" save all the variables in a separate file and then do the get request on that file ") , and the only useful info i found about that was here( jquery - Saving a text file on server using JavaScript - Stack Overflow)(but i think it overcomplicates things).

I don't quite understand what you mean when you mention PHP. In what i have been able to achieve so far (obtain the html page) i haven't used any php. So could you expand on that?

The getting of web pages is easy, as you see you just get the full text body.

The problem you have is that you do not actually have a clear line of data.
It seems to be created or retrieved by the JavaScript that runs on the browser.
The ESP or Arduino can't do much with the JavaScript apart from read the words.

PHP is ServerSide execution, allowing you to have far more control over the output data.
It also allows you to gather all the data you need, even from other web pages, manipulate it if needed and then output it in a format you want or expect. Any worthwhile hosting site should allow you PHP.

<?php

t="Hello";
echo t;
exit;

?>

If you are simply relying on the data you are getting presented and have no control of it's format then you just need to write the ESP code to deal with it, but if you can actually format and present the data as you want it before it hits the ESP then even easier.

ok, so lets say i have everything setup. a web site on the internet with the string that i want.

<!DOCTYPE html>
<html>
  <body>

    <?php
      t="hello";
      echo t;
      exit;
    ?>

  </body>
</html>

and the arduino now prints:

200
<!DOCTYPE html>
<html>
  <body>

    <?php
      t="hello";
      echo t;
      exit;
    ?>

  <meta name='forge-tag' value='forge-token:1564152950' /></body>
</html>

so, is it not working how it is supposed to or am i missing something?
(and don't variables in php need dollar signs?)
(i still uploaded it to the same web hosting site http://hamster-194.getforge.io/)

Humm,
Well you can try and rename the file extension to ".php"
Guessing it is index.html or similar right now.

Or just create a test.php file with the echo test and see if you can /get it on the esp.
if you just get the text again then PHP is not enabled on the server

And I think you are right $t, though it maybe a soft condition, I forget.

if i change the index.html file to a .php file, the web hosting site can't deploy it, i'm guessing because it is just program to look for the index.html file to open it.
And if i have a separate php file, and than include it in the index.html (include("phpfile.php") the esp prints the same php code in the body of the html.
but i understand what you are trying to do, with the echo comand the string gets written in the html code itself, so when i call the get request on the esp8266 and the html code is obtained, i can see the string in the message (right?).
I think i'll try to find another way to host this website (maybe also with a local server on my pc?), than i'll return here for my original question.

Yep it sounds like you do not get to have PHP on that server.
You only need it if you cannot get the output you need from the received response.

I did not really see where your "data" was coming from, but you are reading the response to the request so you can tick that part off the list. If you can find the server data you need and output it as a readable string you're 90% of the way.

Best of luck, drop back let us know how it goes or shout if you need more.