[Solved] Nano - Data input from several serials

Hello Community,

I am new to the Arduino world, and I am well started into a project using a Nano and data inputs from 2 serial devices:

  • ESP8266 fetching a JSON string on the net, wired on RX0.
  • HMC5883 getting magnetic field used for a compass, wired on SDA/SCL.

I won't detail my project in full length as most information will not be relevant for my question. I'm basically using the parsed data to move motors and display information on LCD.

I have developed and run my full program successfully on a MEGA 2560 board on the same ports, and thus felt confident in my attempt to copy/paste the sketch onto my Nano. But to no avail.

Data from HMC compass seem to work OK, but the JSON string from ESP is polluted and unreadable. I am suspecting both data inputs are getting mixed up in one unreadable gibberish.

HMC is throwing out data at something around 20HZ, since the information needs to be updated in "real time", while I set the ESP to send data every 3 seconds.

I have tried to cool things down with two techniques:

  • Interrupt on RX0
  • Serial.Event for RX0 data read

These require to slice the data into single char, and concat the chars until the end of the line is reached. The functions do work as the data is parsed only when '\n' is read, but it doesn't solve my problem which is that weird characters are inserted inside the data before reaching '\n'.

My question is: is it possible to make the board focus on the reading of one serial only? Data from compass is only needed in set up loop, is it possible to disable it in main loop? How come the sketch is working like a charm on MEGA and not on Nano?

Here's a simplified sketch that roughly explains the issue:

#include <Wire.h>
#include <ArduinoJson.h>
#include <DFRobot_QMC5883.h>
String payload="";
DFRobot_QMC5883 compass;

void setup() {
  //*********************************compass initialization****************************
  Serial.begin(115200);
  
  while (!compass.begin()) {
    Serial.println("Could not find a valid 5883 sensor, check wiring!");
    delay(500);
  }
  Serial.println("Initialize HMC5883");
    // compass.setRange(HMC5883L_RANGE_1_3GA);
    // compass.setMeasurementMode(HMC5883L_CONTINOUS);
  compass.setDataRate(HMC5883L_DATARATE_15HZ);
    // compass.setSamples(HMC5883L_SAMPLES_8);

  delay(1000);
}

void loop() {
 
  while (Serial.available() > 0) {
    payload = Serial.readString();
    Serial.print(payload);

    char json[500];
    payload.toCharArray(json, 500);
    StaticJsonDocument<200> doc;

    DeserializationError err = deserializeJson(doc, json);

    if (err) {
      Serial.print("ERROR: ");
      Serial.println(err.c_str());
      return;
    }
    const char *jsonlongitude = doc["iss_position"]["longitude"]; // "64.6418"
    const char *jsonlatitude = doc["iss_position"]["latitude"];   // "-51.6235"
    float longitude2 = atof(jsonlongitude);
    float latitude2 = atof(jsonlatitude);
    while (!Serial)
      continue;

    Serial.print("longitude: ");
    Serial.println(longitude2, 4);
    Serial.print("latitude: ");
    Serial.println(latitude2, 4);
    delay(1000);
    //*********************************compass*******************************
    float declinationAngle = (4.0 + (26.0 / 60.0)) / (180 / PI);
    compass.setDeclinationAngle(declinationAngle);
    Vector mag = compass.readRaw();
    compass.getHeadingDegrees();
    Serial.print("X:");
    Serial.print(mag.XAxis);
    Serial.print(" Y:");
    Serial.print(mag.YAxis);
    Serial.print(" Z:");
    Serial.println(mag.ZAxis);
    Serial.print("Degress = ");
    Serial.println(mag.HeadingDegress);
    delay(1000);
}

Thanks for your help!
Etienne

Welcome to the forum.

Here is what I found about your sketch:

  • You are using delay(). delay() is wasting CPU cycles and stops your sketch from doing anything else. It is a bad software practices and will prevent you from extending your sketch without modifying what you already have. Have a look at the following example to learn how to time parts of your code without delay().

File -> Examples -> 02.Digital -> BlinkWithoutDelay

  • You are using while(). This will create similar issues as delay(). You should try to make your loop() function run as often as possible. Do not stop for anything. Use state variables and flags to remember where you are and do other things. This will allow your Arduino to do many things at the same time.

  • You could improve your variable naming. How do you keep track of what longitude2 means?

  • Are you using Serial as connection to Serial Monitor and another device?
    While you might be able to get away with that when you use TX to Serial Monitor and RX from the device only, you should make that clear in your code. Usually, one would expect a serial connection to be connected to on device only. You can use #define to add an alias and use that for all RX code. When you later find you have another UART available you can change that with minimal effort.

Check out this post for a Mega to ESP8266 example code using json
Also check out the circuits here for connections between Uno/Mega and ESP32 Serial1,Serial2 which leaves usb available
The Uno connection uses Software Serial so its USB is also available.
The code in the post is a revised version of the webpage code.

Hi, had personal issues sorry for not replying.
Now these are hints, thanks a bunch for looking into my problem.

Delay() were used in total despair, and I noticed the sketch was working a lil better. But this obviously didn't fix the root cause, so I've deleted them.

I went back to using a serialEvent to prioritize the ESP communication over the HMC, and also get rid of the while() in the main loop. Here's my new sketch:

#include <ArduinoJson.h>
#include <ArduinoJson.hpp>
#include <DFRobot_QMC5883.h>
#include <Wire.h>
String payload = "";
DFRobot_QMC5883 compass;
bool stringcomplete = false;
char json[500];

void setup() {
  //*********************************compass*******************************
  Serial.begin(115200);

  while (!compass.begin()) {
    Serial.println("Could not find a valid 5883 sensor, check wiring!");
    delay(500);
  }
  Serial.println("Initialize HMC5883");

}
void loop() {

  if (stringcomplete) {

    StaticJsonDocument<200> doc;

    DeserializationError err = deserializeJson(doc, json);

    if (err) {
      Serial.print("ERROR: ");
      Serial.println(err.c_str());
      payload = "";
      stringcomplete = false;
      return;
    }
    const char *jsonlongitude = doc["iss_position"]["longitude"]; // "64.6418"
    const char *jsonlatitude = doc["iss_position"]["latitude"];   // "-51.6235"
    float longitude2 = atof(jsonlongitude);
    float latitude2 = atof(jsonlatitude);
    
    Serial.print("longitude: ");
    Serial.println(longitude2, 4);
    Serial.print("latitude: ");
    Serial.println(latitude2, 4);

    payload = "";
    stringcomplete = false;
  }
  //*********************************compass*******************************
  float declinationAngle = (4.0 + (26.0 / 60.0)) / (180 / PI);
  compass.setDeclinationAngle(declinationAngle);
  Vector mag = compass.readRaw();
  compass.getHeadingDegrees();
  Serial.print("X:");
  Serial.print(mag.XAxis);
  Serial.print(" Y:");
  Serial.print(mag.YAxis);
  Serial.print(" Z:");
  Serial.println(mag.ZAxis);
  Serial.print("Degrees = ");
  Serial.println(mag.HeadingDegress);
}

void serialEvent() {
  while (Serial.available() > 0) {
    char inChar = (char)Serial.read();
    payload += inChar;
    if (inChar == '\n') {
      payload.toCharArray(json, 500);
      stringcomplete = true;
    }
  }  
}

When ESP instructions running on their own (when compass instructions disabled), I get perfect information parsing (no error, no corrupted data). When enabling compass, I get this overflow of compass data, with no ESP responding (for your memory, I should get parsing every 3 seconds). Why isn't the serialEvent taking control, isn't it what he's paid for? :rofl:

Not a clue what you are talking about, sorry. If nothing special done from my side, I guess the answer might be no ?

Wow, this is a long thread, with a bunch of different fixes. First I need to understand a bit more where my problem is! But I might go back to it later thanks!

I'm quite confident with my circuit, since ESP is working perfectly on its own, and I'm also using a breadboard adapter.

Forget the SerialEvent it is miss-leading. It just runs at the end of every loop() if there is Serial.available(). It is NOT an interrupt.
In the post I mentioned you will see the software serial for the ESP has a buffer of 256.
If you make the buffer large enough to hold the entire json message you won't loose any of it even if the rest of your loop takes a long time to work with the compass.
Another advantage of the code it that post is that blocks further sends to the ESP until you have had a change to pick up the last one.
Try these two sketches Nano software serial to ESP software serial. I don't have a Nano so not hardware tested.
ESP8266_nano.ino (3.0 KB)

Nano_to_ESP8266.ino (1.6 KB)

:thinking: Hold on hold on hold on, now that I read your sketch I realized something was not clear enough in my post:

I am NOT trying to SEND serialized data from sensors onto a webpage through ESP. I am GETTING a JSON from a webpage (real time coordinates) with ESP01, which then sends full {JSON} to Nano. I have no problem with this process so far.

On the other side, Nano has 2 inputs: JSON from ESP on RX0 pin (every 3s), and compass via SDA/SCL (15Hz).
Both inputs are needed (at their own pace), here's the backbone of my program:

  1. read heading of compass
  2. initialize motor position (so that motor position = compass position at all times in case I move the device)
  3. read object position from web (deserialized longitude and latitude)
  4. move motors to correct heading
  5. repeat every 3s (each time I get fresh JSON from esp)

It's the Nano that's causing me problems, and if you suspect that one buffer gets overloaded, then it is the Nano's.
I'm a noobie, and I understand that maybe my explanations are not cristal clear :woozy_face:


Here’s my layout. It’s pretty straightforward when only esp01 and HMC compass are connected.
Also I need to save digital pins for my motors (5 pins) and my LCD (6 pins) so I’d like to use hardware serial RX0 rather than building a software serial.
Please disregard the other loose wires and the Mega board for now, they are taped on the piece of wood. I know it’s messy but I didn’t want to fiddle around these now.

OK, same problem in the other direction. But messy increasing the RX buffer size on hardware serial and the Nano is memory limited so
suggest you try this
On the ESP8266 side, process the json there and send only the two numbers comma separated. They will fit in a standard 64byte RX buffer.
replace SerialComs coms(...) with just SerialComs com;
That sets up a 60byte message size both ways
Here are a couple of sketches using comma separated values. You will need to add the Json decoding in the ESP code.
NanoCSV.ino (1.8 KB)
ESPtoNanoCSV.ino (2.3 KB)

Before trying out your sketch for smaller data input from ESP, I was trying to improve the version with serialEvent (2nd code shared above). Not that I don't wanna give your solution a try, but a few thoughts popped out of my head.
First, your solution is limiting the size of the data input from the ESP to Nano RX0, but this data input is very limited (128 bytes every 3 seconds) compared to the HMC compass throwing a bunch of numbers at 15Hz on SDA, right?

Anyway, I understood that the serialEvent does not interrupt my code when receiving data, but I like it because it checks that the data is full before trying to compute it. There might be other ways to do that, but I wanted to give it a last shot before tackling anything else.
So, what I'm trying now is to "clean" the payload variable and resetting stringcomplete boolean by repeating the following sequence as much as possible:

payload = "";
stringcomplete = false;

At the end of the loop for instance. Is that a good idea?

Also, something that might interest you. I rarely verify my sketch before uploading it, as the crashes I get so far are undetected. BUT, this time I did, and here's what I got:

Sketch uses 23226 bytes (75%) of program storage space. Maximum is 30720 bytes.
Global variables use 1794 bytes (87%) of dynamic memory, leaving 254 bytes for local variables. Maximum is 2048 bytes.
Low memory available, stability problems may occur.

I don't get this error on my Mega, and for your memory my sketch works on it. Do you think my problem could be related? And in that case, do you think your solution could work?

One possible simple fix from Message Failures - How to fault find and fix it to slow down the Serial baud rate alot, say 300 baud on both the Nano and ESP8266 side. This gives your loop a lot more time to run before the Serial RX buffer overflows.
You realize you are printing the position back to the ESP (via Serial)
I would suggest you go to SoftwareSerial on the Nano for the ESP connection so you can see debug msgs on the monitor but SoftwareSerial turns off interrupts so that could effect your SDA/SCL interface.

Don't think that is a good idea you will clear any accumulated incoming data each loop, before it all arrives.

The NanoCSV.ino above used 534 bytes, so your extra code is using the rest. 254 bytes left is 'ok' provided you don't have any significant method local variables.
I suspect the json[500] is the main source of the memory usage. Drop json and go to CSV format. That is deserialize the Json in the ESP before sending just what you need to the Nano.

The problem with that is that the 128 bytes exceeds the Nano serial RX buffer size so if the loop is slow you will loose data.

My SerialComs program has a number of error messages that let you debug what is failing.
See my latest tutorial, the section on Message Failures - How to fault find and fix
But it is a bit difficult to see them without a USB serial available. Try the code out on a Mega2560 first using Serial1 for the ESP connection and Serial for the SafeString and your debug msgs

Also add a loopTimer, see this tutorial, to see just how long your loop() code is taking to run

let me go back some days: what's the reason to keep all the complexity with serial communication between controllers? How would the system look like, if you just replace the MEGA / NANO by the ESP and do all the processing on one controller - the ESP?

If "Running out of pins" - is the only answer - there are lot of port expanders available or even other ESPs with more pins.

Focusing on one controller, one sketch for one system might make programming far easier and code easier to maintain in future.

@noiasca , true. But.
I went through a bit of docs and forums before starting, and read that ESP 8266 boards such as NodeMCU are not as beginner friendly as pure Arduino boards. So I decided to stick with Nano (for its limited footprint) and ESP01 Wifi-only module, thinking that I would do as minimum work possible on ESP side.

true, until you come to the phrase "pure Arduino boards". You have already tried to connect an ESP with the Arduino. So THIS is the starting point to start thinking about ESP only. It's not so hard - you still use the Arduino IDE and simple sketches will compile and load like for Mega/Nano.

based on your picture above and the code you provided, I assume you are on an experience level to go an ESP-only way. Just the ESP01 is the wrong starting point imho. You mentioned the NodeMCU - good choice. The ESP32 comes with a lot more features (and pins) but also with some more complexitity (and may be that's the reason why one shouldnt start with the esp32).

So it mainly depends what you really need in your final product. If you can make a list of needed interfaces (serials - separarted if you need TX and or RX and at which speed, I2C, SPI, CAN, Wifi...)
Digital Pins (how many in, out)
Analog Pins (how many in, out)
...
one can give you a proposal what controller you should take in consideration.

I’d like to control one stepper motor and one servo. Input is collected from compass and webpage. I basically get position of a star/space object and get motors pointing toward it using magnetic positioning. I also need LCD display and a LED strip that turns ON/OFF if object is visible or not.
Sooooo:

  • Motor + servo + LCD = 4+1+6 = 11 digital outputs
  • Power relay for (optional) LED strip control = 1 digital out.
  • HMC5883 magnetometer = SDA/SCL input
  • Wifi (available API’s on the net usually limit the requests to 1 every 3-4 seconds)

use a I2C Backpack for the LCD (if you solder it to the LCD, it will not take any footprint in your design). Beside to include the right library necessary changes are only minor.

Personally I like the smaller NodeMCU V2 with CP1202 better than the V3 with CN340 because it fits better into a small breadboard, but basically both can be used.

let's see a pinout with some remarks:

pins marked black have some restrictions (e.g. during startup), but overall your project seems to fit on a Nodemcu.

If you don't want to use a LCD I2C Backpack, you are forced to go for an ESP32.
Oh, and just to have it mentioned: the ESPs are 3v3 devices.

@noiasca OK given the limited price of components I might consider your solution if I get really stuck. Replacing the hardware is easy on the paper, but I might run into new deadlocks and I'm not sure I'm ready yet to start over. At this point, I feel that I am this close to finding a solution with what i’ve built up so far.

@drmpf I have to admit, your code scares me, I don't understand half of it.

BUT, I am still trying to get closer to the root cause by placing flags after each step. Here’s what I figured out at this point:

  • It seems that the JSON string IS getting through and is signaled as complete from my serialEvent - most of the times.

  • Some other times, the if() loop gets stuck :

  • I get FULL success when shortening the sketch by removing some unrelated (but functional) and time consuming lines such as servo moves or LCD prints. But of course, that’s not what I want to do.

According to this :[Serial Input Basics - updated - #2 by Robin2] the sketch could be too big for the microchip to process full buffer before more data comes in.
So I guess @drmpf is fully right and his solution might work.

  1. I started by rearranging the if():

To put it in a different way, it’s OK to have an error, but please clear the payload and start again.

  1. I deleted all Serial.print() and now do all debugging with digital and LCD.

  2. I need to deserialize JSON directly on ESP and send a shorter string from ESP to Nano.

Now @drmpf , is it possible to use a function for concatenation with ‘,’ that I can fit in existing code somewhere around here:

And how can I deserialize it afterwards?

And by the way thanks ! :innocent:

Concat is a easy as

Serial.print(longitude2,5); // how may dec do you need printed? here there are 5
Serial.print(',');
Serial.print(latitude2);
Serial.print('\n');   // Serial println() adds \r\n

On the other side you need to read in a line (terminated by \n), i.e. payload

To parse there are a number of ways also. You can use String indexof(',') to find the , and then substring to pick up the first and second number. SafeString has nextToken(token,',') to extract the fields.

The toFloat method is sloppy and just returns 0.0 if there are any errors, so hard to tell if the result is actually 0.0 or some bad data Arduino Software Solutions has a better conversion code

Here is some example code

// download SafeString V4.1.0+ library from the Arduino Library manager or from
// https://www.forward.com.au/pfod/ArduinoProgramming/SafeString/index.html
#include "SafeString.h"

String payload("33.678,55.999");
String payload2("33.678,abc"); // garbage for second number

void decodeCSV(String &csvString) {
  String longStr;
  String latStr;
  float longitude2;
  float latitude2;
  Serial.print("Payload:"); Serial.println(csvString);
  Serial.println(F(" String.toFloat()"));
  int idx = csvString.indexOf(',');
  longStr = csvString.substring(0, idx);
  if (idx > 0) {
    idx++; // set over ,
    latStr = csvString.substring(idx);
  } else {
    Serial.println("Missing second number");
  }
  longitude2 = longStr.toFloat();
  latitude2 = latStr.toFloat();
  if ((longitude2 == 0.0) || (latitude2 == 0.0)) {
    Serial.println(F("likely data errors"));
  }
  Serial.print(F(" longitude2 = ")); Serial.println(longitude2);
  Serial.print(F(" latitude2 = ")); Serial.println(latitude2);

  Serial.println(F("SafeString conversion"));
  cSF(sfCSV, csvString.length()); // make a temporary SafeString to hold the input
  sfCSV = csvString.c_str();  // assign the input to the SafeString
  cSF(field, 20); // to hold numbers
  sfCSV.nextToken(field, ',');
  if (field.toFloat(longitude2)) { // ignores leading and trailing whitespace
    Serial.print(F(" longitude2 = ")); Serial.println(longitude2);
  } else {
    Serial.print(F("Not a valid float '")); Serial.print(field); Serial.println("'");
  }
  sfCSV.nextToken(field, ',');
  if (field.toFloat(latitude2)) { // ignores leading and trailing whitespace
    Serial.print(F(" latitude2 = ")); Serial.println(latitude2);
  } else {
    Serial.print(F("Not a valid float '")); Serial.print(field); Serial.println("'");
  }

}

void setup() {
  Serial.begin(115200);
  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }
  Serial.println();

  decodeCSV(payload);
  Serial.println(F(" ============"));
  decodeCSV(payload2);
}
void loop() {
}

And some output

Payload:33.678,55.999
 String.toFloat()
 longitude2 = 33.68
 latitude2 = 56.00
SafeString conversion
 longitude2 = 33.68
 latitude2 = 56.00
 ============
Payload:33.678,abc
 String.toFloat()
likely data errors
 longitude2 = 33.68
 latitude2 = 0.00
SafeString conversion
 longitude2 = 33.68
Not a valid float 'abc'

The SerialComs methods add a checksum to the message and detects buffer overruns so are more reliable.

OK Problem solved :white_check_mark: thanks to @drmpf.

  • The code on ESP was modified, it is now sending ONLY the needed variables to the Nano separated by ‘,’ in a string, thus freeing buffer space.
  • The parsing is done with your Safestring command when received on the Arduino.
  • Most print() were replaced with print(F()) after seeing it in your sketch and reading about it. Nice touch by the way :+1: .
  • All delay() and Serial.print() were removed.
  • The while() for serial.read() in the loop was also replaced with an if().

I can say that my sketch is now a lot leaner than it was before.

I will try to free some input digital pins by using a serial backpack for LCD as suggested by @noiasca and add a soft keypad to enter manually the observation location (in the code right now). And work on the overall aspect of the machine with nice lights and a fitting case to display on a shelf (yes, another one of these useless and time-consuming Arduino projects!).

I really can’t thank you enough for bearing with me, I have learnt a lot throughout this project!
Étienne