ESP8266 Communication

Helllo!

Today I am trying to figure out the best way to do a project. I am extremely new to internet protocols and would love some pointers on where to start with this:

I have an Arduino Mega currently connected to a sensor and other external indicators. The main thing i'm trying to do here is get the information the sensor reads onto a home server.

My proposal? The ESP8266-01 generic wifi module. I have messed around with this module and have successfully connected it to my home wifi and ran some diagnostic tests to make sure its working (turning a LED on and off with a helpful IoT's website).

At this point I am at a loss of where to continue and where to even look. I have visited the ESP8266 forum and there is a ton of awesome information there, but most of it consists of protocols I don't fully understand and doubt it pertains to my project.

Feasibility wise, am I making this project a lot harder on myself then I need to? If I am on the right track, is there any good starting places to start messing around with programming these ESP8266 modules in a way that receives information from an Arduino Mega and transmit it to server? An example library or something similar to the libraries out there for arduino modules would be perfect, I just need a meaningful way to start actually understanding this code and a way to test.

How I solved this (there may be other ways): - have the ESP8266 craft a POST request with the sensor data, - then on the server side: run a script based on the URL which reads the POST data and puts it in a database. It's a LAMP server, where P stands for Python in my case.

I assume your home server is behind a firewall, so for this to work you'll need your ESP to be on the same network.

If you want to know more about the network protocols involved, check out this article: https://tttapa.github.io/ESP8266/Chap05 - Network Protocols.html
You’ll probably use HTTP to send the data to the server. Chapter 9 explains how HTTP requests work: https://tttapa.github.io/ESP8266/Chap09 - Web Server.html

Installing a LAMP stack is really easy. Here’s a great guide: How To Install Linux, Apache, MySQL, PHP (LAMP) stack on Ubuntu 16.04 | DigitalOcean
I strongly recommend doing this on a Ubuntu computer (or other Linux distro). On Windows, you could use something like XAMP, but it’s not as easy.
I sometimes use my Raspberry Pi as a LAMP server. I prefer Ubuntu Mate, because it’s almost the same as the normal desktop version of Ubuntu.

For server side scripting, I mostly use PHP, because it’s very popular, so there’s a lot of information online.
I use phpMyAdmin to manage my MySQL databases.

Here’s some code I use to take data from a POST request and insert it into the database. Read the links above to learn how to construct such a POST request on the Arduino.
If the script receives a GET request, it just responds with an HTML page containing a table with the database data.

<?php
try { // connect to database
  $db = new PDO('mysql:host=localhost;dbname=IoT;charset=utf8mb4', 
    'MySQL username', 'password', 
    array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION));
} catch(PDOException $ex) {
  http_response_code(500);
  echo "Unable to connect to the database";
  error_log($ex->getMessage());
  die();
}
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
  if (!isset($_POST["time"]) or !isset($_POST["value"])) { // check if request contains 'time' and 'value' data
    http_response_code(400); // bad request
    die();
  }
  try { // insert new values into database
    $stmt = $db->prepare("INSERT INTO sensordata (time, value) VALUES (:t, :v);");
    $stmt->bindValue(':t', $_POST["time"], PDO::PARAM_INT);
    $stmt->bindValue(':v', $_POST["value"], PDO::PARAM_STR);
    $stmt->execute();
  } catch(PDOException $ex) {
    http_response_code(500);
    echo "Can't insert data into database";
    error_log($ex->getMessage());
  }
} elseif ($_SERVER['REQUEST_METHOD'] === 'GET') { // display the table as a webpage
  try {
    echo "<html><body><table>\r\n<tr><th>Index</th><th>Time</th><th>Value</th></tr>\r\n";
    foreach($db->query("SELECT * FROM sensordata") as $row) {
      echo "<tr><td>".$row['id']."</td><td>".$row['time']."</td><td>".$row['value']."</td></tr>\r\n";
    }
    echo "</table>";
  } catch(PDOException $ex) {
    echo "</table>\r\nUnable to retrieve data from database\r\n";
    error_log($ex->getMessage());
  } finally {
    echo "</body></html>";
  }
}
?>

On the Arduino side, you can either use the ESP8266 as a slave, using AT commands.
Or you could program the ESP directly (this is my favorite method). The ESP8266 is a much more powerful microcontroller than the Mega, it has more RAM, runs at a clock speed that is 5 to 10 times higher than most Arduinos, and it has plenty of on-board flash storage.
You can program the ESP to receive sensor data from the Mega, and then do all that is needed to send that data to the server (connect to WiFi, connect to server, construct HTTP request, read response …). Then you don’t have to worry about all that in the Mega code.

If you don’t need many pins for your sensors and other hardware, you could use an ESP8266 dev board (e.g. Wemos or NodeMCU) and leave the Mega out completely. This would be the easiest approach.

Pieter