Go Down

Topic: API Get on Arduino Uno connected to ESP8266 (Read 4180 times) previous topic - next topic

BassoTara

Hi everyone!

My friend and I have to do a University project using Arduino Uno. We have to show daily news titles on a LCD screen connected to Arduino, scroll them and, if interested in the content of a news, we can select it so that Arduino can send it to a mobile application (which is still under development) so we can read it entirely. Arduino is connected to the Internet using ESP8266.

As of now, we managed to connect Arduino to the internet, controlling ESP8266 using AT commands from the Arduino code. We're now trying to do a GET of this API: https://newsapi.org/

If we load news on a simple WebApp, this script is working:

Code: [Select]

$.getJSON("https://newsapi.org/v1/articles?source=the-next-web&sortBy=latest&apiKey=db5debd72b3843699b34b8b838154bde", function(json) {
// Function to show the news we found
}


However, we don't know how to do the same directly on Arduino. We've tried to look it up on the web, but without good results. Do you guys have any advice for us?

PaulS

Quote
Do you guys have any advice for us?
Post the code you have so far.
Explain what it actually does.
Explain how that differs from what you want it to do.
The art of getting good answers lies in asking good questions.

Zardof



As a suggestion.....
Using <WiFiEsp.h>......
Code: [Select]

void httpRequest() {
  // close any connection before send a new request.
  // This will free the socket.
  client.stop();
  // if there's a successful connection:
  if (client.connect(server, 80)) {
    // Serial.println("connecting...");
    client.println("GET /v1/articles?source=the-next-web&sortBy=latest&apiKey={API_KEY}  HTTP/1.1");
    client.println("Host: newsapi.org");
    Serial.println("Host: newsapi.org");
    client.println("User-Agent: ArduinoWiFi/1.1");
    Serial.println("User-Agent: ArduinoWiFi/1.1");
    client.println("Connection: close");
    Serial.println("Connection: close");
    client.println();
  }
  else {
    // if you couldn't make a connection:
    Serial.println("connection failed");
  }



Remember to put your API_KEY.

G

BassoTara

#3
Jul 05, 2017, 05:48 pm Last Edit: Jul 05, 2017, 07:48 pm by BassoTara
Thanks for your suggestion! I managed to get the news and show it (Json-style) on the serial monitor.
Since the Json is really long, I'm not sure if Arduino or the Serial Monitor can't handle it, as it only shows a part. Anyway, thanks for your help, I'll probably need it again so I'll try to go on and if I have problems I'll ask on the forums again.

EDIT: The result we get using the GET method is not the whole JSON String, which should be something like this, but only a part. Is there maximum size for Strings, so that Arduino can't read it completely? Or could it just be a problem on the serial monitor not showing it correctly?

This is the code we've got so far.

Code: [Select]


#include <SoftwareSerial.h>

SoftwareSerial esp8266 (2, 3); //Arduino TX (ESP8266 RX) connected to Arduino Pin 2, Arduino RX(ESP8266 TX) connected to Arduino Pin 3

String ssid = "FASTWEB-1-XXXXXXXXX";
String password = "XXXXXXXXX";

String path = "/v1/articles?source=the-next-web&apiKey=db5debd72b3843699b34b8b838154bde";
String server = "newsapi.org";
String request = "GET " + path + " HTTP/1.1\r\n" + "Host: " + server + "\r\n" + "Connection: keep-alive\r\n\r\n";
String requestLength = String(request.length());

const int timeout = 2000;

void setup() {
  Serial.begin(9600);
  esp8266.begin(9600); // Should match ESP's current baudrate
 
  setupESP8266();
  connectToWiFi();

  startTCPConnection();
  sendGetRequest();
  closeTCPConnection();
}

void loop() {
 
}

String atCommand(String command, int timeout) {
  String response = "";
  esp8266.println(command);

  long int time = millis();

  while( (time+timeout) > millis() ) {
    while(esp8266.available()) {
      char c = esp8266.read();
      response += c;
    }
  }

  Serial.println(response);
  return response;
}

void setupESP8266() {
  atCommand("AT+RST", timeout);
  atCommand("AT+CWMODE=1", timeout);
}

void connectToWiFi() {
  String connect = "AT+CWJAP=\"" +ssid+"\",\"" + password + "\"";
  atCommand(connect, 6000);
  atCommand("AT+CIFSR", timeout);
}

void startTCPConnection() {
  String connect = "AT+CIPSTART=\"TCP\",\"" + server + "\",80";
  atCommand(connect, timeout);
}

void closeTCPConnection() {
  atCommand("AT+CIPCLOSE", timeout);
}

String sendGetRequest() {
  atCommand("AT+CIPSEND=" + requestLength, timeout);
  String response = atCommand(request, 6000);
  return response;
}



Zardof

To read the whole page you would need a lot of memory. Since your code only runs once how do you know your program hasn't crashed? This would explain why you only get part of the web page.

Try a Serial.print at the end of your setup to see if it happens.

G

PieterP

If you're going to be working with large JSON/XML/HTML strings, I'd strongly recommend programming the ESP8266 directly. It has 40 times more RAM than an Arduino UNO, and it can process the data much faster, because its CPU runs 5 to 10 times faster.
It also eliminates the need for cumbersome AT commands.

What kind of ESP8266 module are you using? And what display interface are you using?
The ESP8266 can probably drive the display as well, so you won't need the Arduino at all.

My Beginner's guide to the ESP8266 should get you started.

Pieter

BassoTara

I use this ESP8266 and an LCD 1602 Module display.

Do you recommend directly programming the ESP8266? If so, is there a way I can make it and Arduino Uno communicate, so that the ESP takes care of the Internet part, while the Arduino does the rest?

PieterP

Yes, just program the ESP8266 like a normal Arduino, and then communicate with the Uno over the Serial connection (or I²C).
You can handle all internet-related stuff on the ESP, parse the JSON strings, take out the relevant information, and send it over Serial to the Arduino Uno.
Take a look at Robin2's Serial Input Basics.
The Uno program will be much simpler, because it doesn't have to worry about any of the WiFi stuff, just read the data from the Serial port.

Pieter

PaulRB

#8
Jul 06, 2017, 01:42 pm Last Edit: Jul 06, 2017, 02:04 pm by PaulRB
I use this ESP8266 and an LCD 1602 Module display.

Do you recommend directly programming the ESP8266? If so, is there a way I can make it and Arduino Uno communicate, so that the ESP takes care of the Internet part, while the Arduino does the rest?
You can do it with that esp-01 module, but it would be much easier with an esp8266 Dev board such as Wemos Mini.




With the esp-01 you will need a usb-serial serial adaptor or dedicated esp-01 programmer to upload your sketch:


Does your LCD module have an i2c adaptor backpack? Your esp module could use the LCD directly with one of those.


What does "the rest" mean? What else does the Uno do/have wired to it? The esp-01 can have more I/o attached by using simple i2c adaptors.


Question: how did you intent to supply the esp-01 with 3.3V power? The Uno's 3.3V pin cannot provide enough. You need a regulator like this ams1117 module:

You won't need an extra regulator if using an esp Dev board like the Wemos.

BassoTara

#9
Jul 06, 2017, 02:36 pm Last Edit: Jul 06, 2017, 03:51 pm by BassoTara
To read the whole page you would need a lot of memory. Since your code only runs once how do you know your program hasn't crashed? This would explain why you only get part of the web page.

Try a Serial.print at the end of your setup to see if it happens.

G
I've tried and I can confirm it doesn't crash.

Yes, just program the ESP8266 like a normal Arduino, and then communicate with the Uno over the Serial connection (or I²C).
You can handle all internet-related stuff on the ESP, parse the JSON strings, take out the relevant information, and send it over Serial to the Arduino Uno.
Take a look at Robin2's Serial Input Basics.
The Uno program will be much simpler, because it doesn't have to worry about any of the WiFi stuff, just read the data from the Serial port.

Pieter
Thanks for your tips, I'll try and see how it goes.

You can do it with that esp-01 module, but it would be much easier with an esp8266 Dev board such as Wemos Mini.
I can borrow a NodeMCU from my University, but I would prefer to use the ESP-01, as it is my own, and only borrow the NodeMCU in case I can't do something with my stuff.

With the esp-01 you will need a usb-serial serial adaptor or dedicated esp-01 programmer to upload your sketch
I can upload sketches to the ESP-01 using Arduino Uno (just need to connect the GPIO0 to GND to get into Flash Mode).

Does your LCD module have an i2c adaptor backpack? Your esp module could use the LCD directly with one of those.
No, it doesn't.

What does "the rest" mean? What else does the Uno do/have wired to it? The esp-01 can have more I/o attached by using simple i2c adaptors.
The rest is the LCD display (in case the ESP can't handle it), some buttons to scroll the news, and an Ultrasonic Sensor (which I omitted in the first post because my original problem didn't concern it. Thanks for the tip about the adaptors, I might use one of them.

Question: how did you intent to supply the esp-01 with 3.3V power? The Uno's 3.3V pin cannot provide enough. You need a regulator like this ams1117 module:
You won't need an extra regulator if using an esp Dev board like the Wemos.
For now, I've powered the ESP-01 using the Uno's 3.3V pin; I've read controversial things about this on the Internet, someone says it works, someone says it doesn't. If it doesn't work or is unsafe for the boards, I'll consider using a regulator.

PieterP

I can upload sketches to the ESP-01 using Arduino Uno (just need to connect the GPIO0 to GND to get into Flash Mode).
Correct, just remember to connect TX to TX and RX to RX, and upload an empty sketch to the Arduino first. You also may want to use voltage divider on the RX pin (the UNO has a 1k resistor at the TX line of the ATmega16U2, so you can just use 2k resistor from RX to ground).
Some online sources claim that the ESP's input pins are 5V-tolerant, so it'll probably work without the resistor, but try it at your own risk ;)

For now, I've powered the ESP-01 using the Uno's 3.3V pin; I've read controversial things about this on the Internet, someone says it works, someone says it doesn't. If it doesn't work or is unsafe for the boards, I'll consider using a regulator.
It'll work most of the time, but if you take a look at the voltages on a scope, you can see some scary voltage drops during transmit. If you have decent bypass capacitors, you probably won't notice any serious problems.
For my permanent projects, I always use a more powerful 3.3V LDO, like a 1117-33 or HT7333.

Pieter

PaulRB

The esp-01 can't handle the LCD without the i2c backpack. Those LCDs normally require 6 pins, only 2 with the i2c adaptor.

The i2c I/o extender I mentioned is probably not suitable for use with ultrasonic range sensors. They require careful timing of the echo pulse, and an i2c chip would get in the way of that. You can buy ultrasonic range sensors with an i2c interface built in, but they tend to be more expensive.

I would avoid having to write and debug sketches for two mcus and all the necessary extra code to get them to communicate with each other, especially as they run at different voltages. The whole thing is getting too over complicated. My advice would be to save the Uno for another project. It's a different horse for a different course. Get a Wemos Mini and an i2c adaptor for the lcd display.

BassoTara

Hi again!

We followed Pieter's suggestion and we're now programming the ESP-01. The GET request is working and we're able to parse it successfully.

However, we don't really know what to do to make the ESP communicate with Arduino when they have two different codes (we don't have a code on Arduino yet, we're just trying to start the communication). Most guides on the web seem to assume that only one of the boards is programmed.

Do you guys have any guides or sample sketches we could get some inspiration from?

PieterP

Yes, just program the ESP8266 like a normal Arduino, and then communicate with the Uno over the Serial connection (or I²C).
You can handle all internet-related stuff on the ESP, parse the JSON strings, take out the relevant information, and send it over Serial to the Arduino Uno.
Take a look at Robin2's Serial Input Basics.
The Uno program will be much simpler, because it doesn't have to worry about any of the WiFi stuff, just read the data from the Serial port.
What kind of data do you have to send between the Uno and the ESP?

Pieter

BassoTara

#14
Jul 08, 2017, 03:28 pm Last Edit: Jul 08, 2017, 03:35 pm by BassoTara
We have to send 5-10 parsed strings, which are the news titles, from the ESP to the Uno, then the Uno has to show it on the LCD display. After that, the user can select a piece of news, and the Uno has to send to the ESP its title or array id so that the ESP can recognize it and POST it on another page.

Go Up