Reading several (fast) Serial inputs simultanously on Arduino Mega

Hi,
firstly I want to say, that I have read Robin2's thread Serial Input Basics - updated.

With that said:
I want to read 2 independent HW Serial inputs on Arduino Mega, make some data processing and send the data on Serial ouput (while still receiving new streams of serial data).

I would like to put this question on the Serial Input Basics thread to reach the competent help, but Robin2 advises people with their own projects to start another thread...

The project is to read 2 different sources of LED-matrix with Glediator protocol (serial 1000,000 baud each), compile a new LED image and send it (with Glediator protocol) to the LED-matrix. In terms of performance, I need to read 2 channels of 1536 bytes each, compile a third array of 1536 bytes and send it with 1000,000 baud again. There is no handshaking, so I have to read the incoming streams as they come, probably with no pause in between bytes. Sending of bytes is not critical to pauses, but since I have to send all incoming frames, I have not much chance to put up with pauses in the out stream.

Since I already have tried with no success, I have a couple of questions to serial experts:

  1. Is it ever realistic to read 2 HW serial inputs with such a high baud rate with Arduino Mega?

  2. I have noticed, that I can perform a simple int b = Serial1.read(), followed with Serial1.write(b), but this does not work with Serial. (The Serial.write is "dead" until long time (tens of seconds) after serial bytes end to come to the Serial input.) I understand, that the Serial is somehow different because it needs to distinguish between run time input and programming input (receiving the sketch). Can this behavior be changed? And how can I then come back to the sketch programming?

  3. Since I have a very bad experience with Serial on ESP platform (the sketch reading single Glediator protocol sending it as WS8212B pulses to the LED-matrix worked on Arduino Uno, but not on ESP8266), should I go the ESP way (because of it's high clock frequency)?

  4. Finally, is the following principal the right way to go:

void loop() {
if (Serial1.available()) {
ser1 = Serial1.read();
}
...

if (Serial2.available()) {
ser2 = Serial2.read();
}
...

if (Serial1.availableForWrite()) {
Serial1.write(ser3);
}
...
}

or is there a (much) better way by eg. reading/writing bytes in chunks of 32 ?

tonys_0:
Hi,
firstly I want to say, that I have read Robin2's thread Serial Input Basics - updated.

With that said:
I want to read 2 independent HW Serial inputs on Arduino Mega, make some data processing and send the data on Serial ouput (while still receiving new streams of serial data).

I would like to put this question on the Serial Input Basics thread to reach the competent help, but Robin2 advises people with their own projects to start another thread...

The project is to read 2 different sources of LED-matrix with Glediator protocol (serial 1000,000 baud each), compile a new LED image and send it (with Glediator protocol) to the LED-matrix. In terms of performance, I need to read 2 channels of 1536 bytes each, compile a third array of 1536 bytes and send it with 1000,000 baud again. There is no handshaking, so I have to read the incoming streams as they come, probably with no pause in between bytes. Sending of bytes is not critical to pauses, but since I have to send all incoming frames, I have not much chance to put up with pauses in the out stream.

Since I already have tried with no success, I have a couple of questions to serial experts:

  1. Is it ever realistic to read 2 HW serial inputs with such a high baud rate with Arduino Mega?

  2. I have noticed, that I can perform a simple int b = Serial1.read(), followed with Serial1.write(b), but this does not work with Serial. (The Serial.write is "dead" until long time (tens of seconds) after serial bytes end to come to the Serial input.) I understand, that the Serial is somehow different because it needs to distinguish between run time input and programming input (receiving the sketch). Can this behavior be changed? And how can I then come back to the sketch programming?

  3. Since I have a very bad experience with Serial on ESP platform (the sketch reading single Glediator protocol sending it as WS8212B pulses to the LED-matrix worked on Arduino Uno, but not on ESP8266), should I go the ESP way (because of it's high clock frequency)?

  4. Finally, is the following principal the right way to go:

void loop() {
if (Serial1.available()) {
ser1 = Serial1.read();
}
...

if (Serial2.available()) {
ser2 = Serial2.read();
}
...

if (Serial1.availableForWrite()) {
Serial1.write(ser3);
}
...
}

or is there a (much) better way by eg. reading/writing bytes in chunks of 32 ?

First let me assure you that you do not need to read a all bytes simultaneously. You have the time it takes to receive one byte to remove the previous byte from the USART internal buffer.

What you want to do requires you to write your own serial input/output code your self in assembly. Are capable of doing that and do you have enough time and resources to do that? If not, then find another project.

Paul

tonys_0:
Hi,
firstly I want to say, that I have read Robin2's thread Serial Input Basics - updated.

With that said:
I want to read 2 independent HW Serial inputs on Arduino Mega, make some data processing and send the data on Serial ouput (while still receiving new streams of serial data).

Assuming you want to read from two separate HardwareSerial ports (say Serial and Serial1) then you just need a copy of the appropriate function from my tutorial for each port - with suitable changes to the variable names for one of them. Then the code in loop() could be like this

void loop() {
  recvWithEndMarker();
  recvWithEndMarkerSerial1();

  if (newData == true) {
     // do stuff
  }

  if (newDataSerial1 == true) {
     // do other stuff
  }
}

...R

Hi Robin2,
thank you for your answer. It was my humble hope, that you will read my question.

The "problem" with your suggested solution is the loop:

while (Serial.available() > 0 && newData == false) {

}

that reads all of the Serial input data, before it goes on reading the Serial1 input data.
At least, this is my understanding of the newData flag. You had me to read my question back again to make sure, I did mention that, but it is already in the subject title (simultaneously). My serial data come all the time to Serial and to Serial1, frame after frame... and the frames do not come synchronized. But there is no problem delaying the output stream with the time of a frame period.

As I mentioned, I have read your very appreciated Serial Input Basics article, tried to apply it to my project with no success and my numbered questions (1, 2, 3, 4) were a result of those experiments. Could you be kind to answer them, if you can?

Unfortunately, I will be on a trip coming week, so I cannot follow up with making appropriate experiments, but I will be back at the end of next week.

Thanks again for your time.

Paul_KD7HB:
First let me assure you that you do not need to read a all bytes simultaneously. You have the time it takes to receive one byte to remove the previous byte from the USART internal buffer.

What you want to do requires you to write your own serial input/output code your self in assembly. Are capable of doing that and do you have enough time and resources to do that? If not, then find another project.

Paul

Hi Paul.
It's not, that I am scared of assembly language. In fact, I started with assembly back in 70ties with an Intel 4040 processor, followed by 8080, Z80 (2MHz! clock) etc. And I promise you, that I would be able to write it for the 8080 today. But my hope was, that we have gone a little further since then :slight_smile:
The thing is, that the routine for reading one stream (1000,000 baud) written in plain Arduino C manner works perfectly.

Advising people to find another project could, after all, be good advice, but this is not the reason why this forum was created, is it?

As I understood your project, the input stream from two serial ports, at 100k bytes/second, simultaneously, and one output stream at 100k bytes per second were what you were proposing. With no control of when any input would happen. Possibly some output control.

Am I mistaken? That does not leave much time for your Arduino to process much data.

Paul

tonys_0:
The "problem" with your suggested solution is the loop:

while (Serial.available() > 0 && newData == false) {

}

that reads all of the Serial input data, before it goes on reading the Serial1 input data.

You are not quite correct.

All the WHILE will do is take all the existing bytes from the Input Buffer. That only takes a few microsecs. It does NOT wait for the end of the message. It may take several calls to recvWithEndMarker() before the end of the message has arrived.

If the WHILE was replaced with an IF then you would need to call recvWithEndMarker() once of every byte and that would be slower.

...R

Paul_KD7HB:
As I understood your project, the input stream from two serial ports, at 100k bytes/second, simultaneously, and one output stream at 100k bytes per second were what you were proposing. With no control of when any input would happen. Possibly some output control.

Am I mistaken? That does not leave much time for your Arduino to process much data.

Paul

Hi Paul,
now you are talking.
This was actually my question #1.

I am not doing much processing than replacing some bytes in stream 1 with non zero bytes from stream 2, which is, what I can do while sending the bytes with Serial1. write()

Since I have no idea how the compiler translates this to real code, I cannot estimate the time spent on each task. That's why I ask people, who know more. Some initial testing showed dropping receiving bytes.
If that's a problem even reading bytes in chunks, I might use ESP platform instead - which was what I would expect people like you would suggest :slight_smile:

Robin2:
You are not quite correct.

All the WHILE will do is take all the existing bytes from the Input Buffer. That only takes a few microsecs. It does NOT wait for the end of the message. It may take several calls to recvWithEndMarker() before the end of the message has arrived.

If the WHILE was replaced with an IF then you would need to call recvWithEndMarker() once of every byte and that would be slower.

...R

Thanks Robin2,
I will do the testing when I am back...

You're trying to process 300K characters a second on a 16MHz processor. That gives you about fifty clock cycles per character - not much time, which I suspect is why Paul suggested the use of assembler.

I'd buy my way out of trouble and get some faster hardware, a Teensy perhaps.

Well, finally I have made use of many of your pieces of advice and I have a functioning solution.

  1. I thank you wildbill for directing me towards higher processor clock frequency, though I cannot afford the Teensy, so I ended up with ESP32 (240MHz). Firstly I hated that thing because of pretending to be like other Arduinos. It's NOT. It's not even a better brother to ESP8266 as they often claim. It's something in its own class and until you get proper documentation, do not bother to use it. What really helped to solve my problem were the 2 cores of ESP32, that can run two tasks simultaneously (if you know, how to do it).

  2. I thank you Paul for directing me to lower language. I did not have to go all the way down to assembler, but I had to go down to Espessifs own APIs. I used them in Arduino IDE and I was lucky they worked there (they are supposed to be used in ESP-IDF programming tool - which is much more complicated and the programing-testing cycle is much longer). The Arduino Serial1.begin, Serial1.readBytes(), Serial1.write() concept is working for ESP32, but is notoriously slower than the native APIs.

  3. I thank you Robin for explanation of the "while" mechanism when you actualy think "if". It helped me to accept the logic and after many days of testing in the end it made me understand, that Arduino Mega will never have a chance to do these 3 things fast enough, not mention the reliability.

So, in the end, I owe you all to say, what the project is about.

There are solutions for controlling homemade LED-matrixes (made out of LED-stripes).
One of them is Glediator, another is Jinx!
They are free, but they are not open source. You can use them, but there are always small features, that you would like to implement or modify but you can't.

What I wanted was to use Glediators fantastic abilities to compose graphics animation effects. When scrolling text, the Glediator has a strange thing, that many people would like to be without. It scrolls the text forwards and back!!! and you can not turn this off. So for the scrolling text, I wanted to use the Jinx! With other words, I wanted to scroll the Jinx! text over the Glediators background graphics animation.
They both have a Glediator protocol serial output. You can run them both simultaneously on a PC and have two USB devices to convert to serial TTL outputs (1000 000 baud each).
So what I did was to build an ESP32 device, that takes in those 2 serial (not synchronized!) streams, writes over the Jinx! text over the Glediator graphics and sends the result as Glediator protocol to the LED-matrix (which already has an Arduino Pro Mini, that converts it to time critical sequence for WS8212B LED-strips).

Now somebody can say, that I should instead redirect the serial outputs from PC, blablabla...

Well, this is the way I was able to solve it. Next time I will create the scrolling (or bouncing) text in a Linux machine or a WiFi connected Arduino with Glediator protocol output - so I can simply use the same ESP32 solution.

So again thank you all, who are still reading this :slight_smile:

Thanks for the update. Many times I wonder if a poster ever finished a project. Glad it is working for you. Don't try to make it better,

Paul

tonys_0:

  1. I thank you wildbill for directing me towards higher processor clock frequency, though I cannot afford the Teensy

I don't understand that statement. A Teensy 3.2 will set you back less than $20. For that you get a 32-bit ARM processor that runs at 6x the clock speed of your 8-bit AVR Mega ($30). And, IMO, it's documented much more coherently than the ESP family.

tonys_0:
so I ended up with ESP32 (240MHz). Firstly I hated that thing because of pretending to be like other Arduinos. It's NOT.

That's a ridiculous statement. The ESP doesn't "pretend" to be anything. It is what it is. A group of dedicated enthusiast created the core BSP files necessary to use it in the Arduino ecosystem. They did a pretty good job, IMO. But, you don't have to use Arduino to program it, there are other IDEs / Tool Chains available. In fact, you don't even have to use it at all. As far as I can tell, you we're forced.

Hi @gfvalvo,
sorry to hurt your feelings.
If I would not see the “Arduino ESP” label so often, I would probably never stumble over it nor order it from China - which at the same time explains, why I can not afford Teensy :slight_smile:

What really appeals to me is the ease with which you can program the Arduino IDE. I am talking about the developing cycle including the ease of uploading. The dedicated enthusiasts deserve all my credit for their work. I had extreme luck that the Espressif APIs worked under the Arduino IDE, because I would never dare to switch to ESP-IDF, which you indirectly suggest.