Arduino Mega send data to ESP8266-01 then ESP forward to RPI (MQTT)

I have an ESP8266-01 connected to an Arduino Mega via TX and RX. My ESP has been flashed so AT commands won't work. The ESP has a simple MQTT code that as of right now is sending "Hello World from ESP_MQTT_PUB_SUB......" to my RPI 3 over local wifi network. The Arduino does nothing at this time because I can't figure out (understand) a way to send the data from the arduino to my ESP and then have the ESP send that data to my RPI. Most of the stuff that I see uses Ethernet or going out of the local network to a web page, NEITHER are what I want to do. I'm guessing that the ESP is both a server and client (PUB_SUB code) because it sends messages and at the same time can receive from my other RPI.

Just to be clear, I'm not looking for someone to write a sketch for me, just help me understand how.
My final plan is to have my ESP connected (via wired to TX, RX) to my Arduino Mega that has a 2x16 LCD screen, 10 water solenoids, 10 Hygrometers, 3 DHT22s, & a rain detector in my garden (about 30 feet from my wireless router) and send the data that it receives through my local network inside my house to my RPI (about 15 from the router the opposite direction) and display the data on my screen. But right now I'm just asking for help with retrieving and forwarding data as it comes available.
NOTE: These are not my actual codes, I use them for testing and learning.

Here is the test code that I'm using for the ESP8266-01 :

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

const char* ssid = "MyRouter";
const char* password = "MyPsswd";
const char* mqtt_server = "MyPi3IP";
#define mqtt_user "Username"
#define mqtt_password "Password"

WiFiClient espClient;
PubSubClient client(espClient);
long lastMsg = 0;
char msg[50];
int value = 0;

void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  randomSeed(micros());

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection using ESP_MQTT-PUB-SUB...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (client.connect("ESP8266Client", mqtt_user, mqtt_password)) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      client.publish("outTopic", "hello world");
      // ... and resubscribe
      client.subscribe("inTopic");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void setup() {
  pinMode(BUILTIN_LED, OUTPUT);     // Initialize the BUILTIN_LED pin as an output
  Serial.begin(115200);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
  client.setCallback(callback);
}

void loop() {

  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  long now = millis();
  if (now - lastMsg > 2000) {
    lastMsg = now;
    ++value;
    snprintf (msg, 75, "Hello World from ESP_MQTT_PUB_SUB #%ld", value);
    Serial.print("Publish message: ");
    Serial.println(msg);
    client.publish("outTopic", msg);
  }
}

Here is the test code that I'm using for the DHT sensor on my Arduino Uno:

#include <Adafruit_Sensor.h>
#include <DHT.h>;

//Constants
#define DHTPIN 2     // what pin we're connected to
#define DHTTYPE DHT11   // DHT 11  (AM2302)
DHT dht(DHTPIN, DHTTYPE); //// Initialize DHT sensor for normal 16mhz Arduino

float hum;  //Stores humidity value
float temp; //Stores temperature value

void setup() {
  Serial.begin(115200);
  dht.begin();
  }

void loop() {
 
  /////////////////////////////////DHT11///////////////////////////////
  {
    Serial.println("     Temperature and Humidity (DHT22)");
    delay(1000);
    hum = dht.readHumidity();
    temp = dht.readTemperature() * 1.8 + 32.1;
    Serial.print("Humidity = ");
    Serial.print(hum);
    Serial.println(" %");
    Serial.print("Temp = ");
    Serial.print(temp);
    Serial.println(" degrees Fahrenheit");
    Serial.println("");
    Serial.println("");
    delay(2000);       //  waits 2000 milliseconds (2 sec).
  }

  }

Why not cut out the Arduino altogether and connect the DHT11 directly to the ESP-01?

Because I want to learn and in my garden I have 2 DHT22’s, 10 Hygrometers, 10 water solenoids, a rain sensor, and an lcd screen all connected to an Arduino mega.

Why not just use Serial?
http://forum.arduino.cc/index.php?topic=396450.0

I don't understand how that will work with my ESP (having one code)grabbing data from my Arduino (which has a different code) and forwarding it over to my RPI. Not trying to be difficult but I honestly don't understand.

Maybe you should start by clearly and concisely stating what you want to do. Most importantly, over what channels / medium do you envision these various data transfers taking place?

Of course the ESP and Arduino will have different programs running. They appear to be performing different functions. You need to write code in one to send data (via some channel /medium) and receive data in the other. Two different functions, two different codes.

Not sure there has to be any “grabbing” (not even sure what that means), just sending and receiving.

oibdrew:
I don't understand how that will work with my ESP (having one code)grabbing data from my Arduino (which has a different code) and forwarding it over to my RPI. Not trying to be difficult but I honestly don't understand.

When used as a serial device, the ESP uses AT command firmware.

The ESP has been flashed, AT commands wont work.

aarg:
So reflash it.

Why? It seems to me what OP wants is for ESP to receive data from the Mega and post it somewhere over WiFi.

What is STILL unclear is how OP wants to do the data transfer from Mega to ESP. Over what channel?

gfvalvo:
Why? It seems to me what OP wants is for ESP to receive data from the Mega and post it somewhere over WiFi.

What is STILL unclear is how OP wants to do that data transfer.

It's just good to understand that it is possible.

oibdrew:
gfvalvo, what are you referring to when you say "OP"?

OP means Original Poster -- that would be you.

It can also mean Original Post -- that would be your first post for this topic

What is so hard to understand that an Arduino is getting data and an ESP is connected to it(wired tx and rx) acting as a "middle man" between it(the Arduino) and a RPI?

What's so hard to understand is that just now is the FIRST time that you mentioned that the Arduino and ESP are wired together!!! I asked you back in Reply #6 -- "over what channels / medium do you envision these various data transfers taking place?", and it's taken you to Reply #12 to tell me " wired tx and rx".

Good engineering starts with writing a complete and comprehensible description of the problem. You completely failed to provide that.

So, now I can tell you to go back to my suggestion in Reply #4. Use Serial communication. Read the tutorial I linked. I believe the Mega has multiple Hardware Serial ports (UARTs) and the ESP can user Software Serial on several of its pins. Write a simple communications protocol to send data from the Mega and receive it on the ESP. Off the top of my head, the protocol should probably send data in Sensor ID / Sensor Value pairs with some kind of delimiter between them and between each pair.

Seems pretty straight forward.

oibdrew:
How do I delete this post because it never got solved? I would like to revise my question so that it is better understood and I can get it answered.

No need to delete, it's better to just revise or clarify your question, and post it here again. If you want, you can edit the thread title. It is best not to edit previous posts, to avoid confusion.

Is this still not understandable?

Serial seems like the way to go (Reply #3). Is that not working for you? If not, why? Post your code and explain what the problem is.

The main problem is that I've been trying so hard for months and I've only gotten this far. The ESP is so simple to use for most but I'm struggling to understand how to add it to my project. I've tried so many things that I'm confused as to what I've tried and what to try next. It's really consuming my life and making me very irritated. I agree from what I've read that Software Serial is what I need to use. The code that I originally posted for the ESP-01 is the same. I've tried many example codes on my Arduino and edited them to test but haven't found one that helps me understand how it works. Most use Ethernet.h so Ive changed those to use WiFi.h. I can see in my head how it should go. Arduino reads value from DHT and outputs it into software serial, from there it should go to my ESP-01 (Arduino tx to ESP rx (I've tried tx to tx and rx to rx as well)). The Esp in turn takes the data and sends it to the RPI over my local network where the RPI displays it in the cmd line terminal.

Is it possible to edit the code that is on the ESP and use it on the Arduino? Similar codes but original on ESP and edited on Uno. I'm playing around with it and seeing if I can't get some communication between the two.

First off, is your Arduino an Uno or a Mega? -- You claim both in different parts of this thread. If it's a Mega, you won't need Software Serial on it because it has multiple hardware UARTS. Just use one of the hardware Serials that isn't for the USB / software download / Serial Monitor. If it's an Uno, then I'd use Software Serial on GPIO so as not to interfere with the above functions.

For the ESP, your use of ESP8266-01 really limits you. The 8266 is very limited on GPIO to begin with and it looks like the -01 is even more limited (not totally sure, I've never used it). I use this Feather-format ESP8266 board from Adafruit:

At least it gives you some GPIO options for your serial interface. The 8266 only has one hardware UART, and I'd avoid using it for this comm channel due to its other uses. So you're stuck with Software Serial on available GPIO pins.

If this were my project, I'd start out very simple -- scrap all the senor readings, MQTT, etc and just concentrate on getting the Serial comm channel working. Use hardware Serial 1, 2, or 3 on the Mega (or Software Serial if Uno) on the Arduino and Software Serial on the ESP. Set up a bidirectional link, TX on one board goes to RX on the other board.

Now, read and understand the tutorial I linked in Reply #3. Get simple comm going between the boards. After that, add your sensors to the Arduino code and upgrade the code on each end to send sensor data from Arduino and receive it on ESP. Confirm the sensor reading are correctly received in the ESP. Finally, add the MQTT code to post the results.

EDIT:
Just recalled that Uno and Mega are 5V logic. ESP is 3.3. You might need some level translation.

"I have an ESP8266-01 connected to an Arduino Uno via TX and RX (Uno is used for testing)."
"My final plan is to have my ESP connected (via wired to TX, RX) to my Arduino Mega......"
I am only learning and testing with an Uno but I have a Mega running outside in my garden. I don't know what UARTS mean but I will look it up. I did not realize that I couldn't do the same communication on my Uno as the Mega, that will make the learning even slower.

I am working with the ESP-01 because that is what I have and for now I'd like to learn how to use it instead of buying something else. I feel that it is good for what I want from it and that is just to pass data over local network. I most certainly could be wrong but I am learning.
I did get a D1 Mini NodeMCU ESP8266-12 WeMos but I haven't done anything with it yet.

I am just trying to learn how to get communication working without the sensors and such, that's why I use my Uno.

I started writing a sketch and actually got it to work briefly (passed a message from the arduino sketch to the ESP (connected tx to rx) and then from the ESP to my pi over my local wifi network) but could not repeat for some reason. I'm getting close! Thank you for your guidance and advice.

gfvalvo:
]EDIT:[/u][/b]
Just recalled that Uno and Mega are 5V logic. ESP is 3.3. You might need some level translation.

I've messing with the ESP for months so I'm aware of it being 3.3 v dependent.

I got another Mega so I can learn with it. Maybe I'll find some help on what I want to do with it.

I still need help.
I have an arduino mega that I'm using for my testing now. If I have my rx from my esp connected to the tx of my arduino and the tx from esp to rx of the arduino, how do I have the esp read the data from my arduino and send it from there. Its the whole reading and forwarding the data that I can't understand.

somebody please help me