Software Serial Communication Text Artifacts / Strange Behavior

Using Arduino Programming Questions
1

I am having trouble with a Hardware Serial to Software Serial program I have written, and after 8+ hours of googling, forum surfing, and attempts to fix, I'm hoping for a fresh pair of more experienced eyes.

I admittedly have just gotten into Arduino, and only have a little C++ knowledge for a HS class I took back in 2002.

I am trying to us and Arduino to talk via console port to a Cisco Switch. "most" of the time it works, but I've noticed some artifacts in transmission, and some strange behavior that I'm just not getting.

Sometimes I get artifacts, seen in Yellow, and often I'm getting double output back, which I'm not really sure why.

At this point I'm at the guess phase where I'm just grasping at straws trying things without really understanding fully what I'm trying.

Can anyone help me understand this a little better?

#include <SoftwareSerial.h>

void printSoftToHard();
void printHardToSoft();

SoftwareSerial mySerial(10, 9); 
 
String messageRead;
String messageWrite;

void setup() {
  
  Serial.begin(9600);
  mySerial.begin(9600);

}

void loop() {
printSoftToHard();
printHardToSoft();
}

void printSoftToHard() {

  mySerial.listen();
  messageRead = mySerial.readString(); 
  if (messageRead != ""){ 
  Serial.print(messageRead); }
  messageRead = "";
}

void printHardToSoft() {
  
  messageWrite = Serial.readString(); 
  if (messageWrite != ""){ 
  mySerial.print(messageWrite); }
  messageWrite = "";
}

poorSerial.png
poorSerial.png1296×800 48.9 KB

2

With one instance of SoftwareSerial, it is not necessary to call listen().

Since you are not actually doing anything with the data, pissing away resources using the String class is not necessary.

3

Is it wise to read from serial even when nothing is available to be read ?

4

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data.

...R

5

Robin2:
Have a look at the examples in Serial Input Basics - simple reliable ways to receive data.

...R

Thanks for the Advice all. I've looked at the examples Robin, and I've modified some of the code there with mine. This have been renamed so I know what they are doing, and I know there is some bloat. Its there now just to make it easier for me to read and keep track of.

6

Apparently I can only post once every 5 minutes lol.

So with the Code below for reading the software serial, i'm still getting a somewhat similar issue, I get occasional character drops when receiving data to the arduino, and some other artifacts still.

I did some more reading on the Switch itself, its days it uses xon xoff, which i'm not quite sure how to implement in Ardunio to test. Is it possible that some of the artifacts and misc characters are control information its sending while I'm reading from it?

#include <SoftwareSerial.h>
SoftwareSerial mySerial(10, 9);


void dislplaySoftRx();
void softRxWithEndMarker();

void sendHardTx();
void readHardTxWithEndMarker();

const int numChars = 512;
char receivingChars[numChars];  
char sendingChars[numChars];  

boolean newRxData = false;
boolean newTxData = false;

void setup() {
 Serial.begin(9600);
 mySerial.begin(9600);
 Serial.println("Reading....");
 mySerial.println();
}

void loop() {
 
 softRxWithEndMarker();
 dislplaySoftRx();
 
 readHardTxWithEndMarker();
 sendHardTx();
 
}

void softRxWithEndMarker() {
 static byte ndx = 0;
 char endMarker = '\n';
 char rc;

 if (mySerial.available() > 0) {
     while (mySerial.available() > 0 && newRxData == false) {
       rc = mySerial.read();
       if (rc != endMarker) {
         receivingChars[ndx] = rc;
         ndx++;
         if (ndx >= numChars) {
           ndx = numChars - 1;
         }
       }
       else {
         receivingChars[ndx] = '\0'; // terminate the string
         ndx = 0;
         newRxData = true;
       }
     }
 }
}

void dislplaySoftRx() {
 if (newRxData == true) {
   Serial.println(receivingChars);
   newRxData = false;
 }
}


void readHardTxWithEndMarker() {
 static byte ndx = 0;
 char endMarker = '\n';
 char rc;
 
  if (Serial.available() > 0) {
     while (Serial.available() > 0 && newTxData == false) {
       rc = Serial.read();
       if (rc != endMarker) {
         sendingChars[ndx] = rc;
         ndx++;
         if (ndx >= numChars) {
           ndx = numChars - 1;
         }
       }
               
       else {
         sendingChars[ndx] = '\0'; // terminate the string
         ndx = 0;
         newTxData = true;
       }
   }
 }
}


void sendHardTx() {
 if (newTxData == true) {
   mySerial.println(sendingChars);
   newTxData = false;
 }
}

poorSerial2.png
poorSerial2.png765×689 9.05 KB

7

I wonder if you aren't running into buffer issues.

Both Hardware and Software serial each of 64-byte buffers for TX and RX. Why are you using 512-byte buffers on top of that? (Which, by the way, consume 1/2 of the RAM available on an Uno.)

Why not just TX/RX byte for byte as you receive them?

If you insist on using your huge buffers, I would make use of Serial.flush() to make sure the transmit buffer is empty before moving on from your Serial.print().

8

Lol, because I'm a noob. I don't know the language really, and other then reading it logically, I don't really fully understand how it all works. I noticed that for some large text prints from the switch, that it would miss information. I could figure out it if was an issue where I just wasn't recieving it, or if there was a buffer that was filling before I could write out. Like I said before, I'm just trying to learn and get it going, then I can start worrying about making it more functional.

Don't worry, I knew it was using up a ton of ram on me, I was already thinking if that did fix the issue I'd need to look at writing the data out, or using strings with F() or something. I just want it to work well, this is the hardest part. The idea here is to have a wifi, and bluetooth console link. I just had the idea one day, and arduino looked like the way to go, and at 5$ for a clone how could you pass it up. That was 3 weeks ago.

9600 baud is so slow, I can't imagine somehow the arduino can't receive the data fast enough, and I tried putting a 2 ms delay on the char read function incase it was actually going too fast or something.

I don't think it would be a buffer issue, because it there are artifacts with such small amounts of data, but that is why I was messing with that variable in sram.

But I just got that max3232 breakout this weekend and have been obsessed with getting this working.

Anyway, thanks for the help and Idea's, and if there is information you think I need to pour over that could help link away.

Thanks again all, this community seems really active its pretty awesome. This Arduino this is awesome too, most fun I've had since Lego's :smiley:

9

mrm0xie:
I don't know the language really, and other then reading it logically

Unlike writing a high-level utility on a PC, you are directly programming registers, timers, and other devices in the micrcontroller. So sometimes you have to think one step deeper.

mrm0xie:
The idea here is to have a wifi, and bluetooth console link.

WiFi (Ethernet) and Bluetooth are protocol heavy interfaces. Don't cheap out and get low-cost PHY chips. That'll be too much work for the poor 8-bit AVR to do it.

mrm0xie:
9600 baud is so slow, I can't imagine somehow the arduino can't receive the data fast enough

Emulating serial in software relies on change interrupts to detect bit changes to bit-bang serial data into the device. Software Serial tends to work well for transmitting to something, but can relatively easily lose data on receive.

If a single bit gets missed, then the software serial port will end up with corrupted data.

mrm0xie:
I tried putting a 2 ms delay on the char read function

That delays when you move the character from the receive buffer to something else (a variable). So delaying the read means you are buffering even more data. Probably not what you intended.

Here's what I'd suggest:

  1. Up the hardware serial port to a faster speed like 115200. Get its buffers emptied as fast as possible.

  2. Look at the Software Serial Example. It shows how to send/receive characters without using additional buffers. Adding buffers is just going to cause more overhead and less time to watch the software serial port.

10

I'll go as far as to suggest to ditch Software Serial completely.

I mean these days you can get a Mega clone from China for < $15 (or if you want to support a vendor, how about a $12 Teensy LC)?

Or even a Leonardo (or Arduino Micro for breadboard use) which allows a connection to the Serial port and still have USB I/O.

All of those solutions give you more than a single hardware serial port (well the 32u4 solutions only give you 1x serial plus USB but in your case I think it's good enough).

Just a thought.

Regards,

Brad.

11

mrm0xie:
I did some more reading on the Switch itself, its days it uses xon xoff, which i'm not quite sure how to implement in Ardunio to test. Is it possible that some of the artifacts and misc characters are control information its sending while I'm reading from it?

I doubt the switch sends the control characters for xon/xoff; it will only do so if you send a large amount of data to it and its buffer is getting full.

Your Arduino is (in this case) the receiving end and can control the flow using XON/XOFF.

This is the theoretical approach:

  • Once your buffer is filled up to e.g. 16 bytes, you send a XOFF (0x13) to the sender; this will stop the sender from sending data.
  • Once you have processed your 16 bytes, you send a XON (0x11) to the sender; the sender will now resume sending data.
  • Go back to (1)

I have never used software serial (does it have a write method?) and I have never used xon/xoff, but you can give it a shot.

12

mrm0xie:
So with the Code below for reading the software serial, i'm still getting a somewhat similar issue, I get occasional character drops when receiving data to the arduino, and some other artifacts still.

When you run into problems the first thing to do is simplify.

Take one of the receive functions out of the program altogether and concentrate on getting the remaining one to work perfectly.

Can you give an example of the data you are trying to receive and an example of the problem that you get.

I don't see why SoftwareSerial should not work properly.

...R

13

sterretje:
I doubt the switch sends the control characters for xon/xoff; it will only do so if you send a large amount of data to it and its buffer is getting full.

Your Arduino is (in this case) the receiving end and can control the flow using XON/XOFF.

This is the theoretical approach:

  • Once your buffer is filled up to e.g. 16 bytes, you send a XOFF (0x13) to the sender; this will stop the sender from sending data.
  • Once you have processed your 16 bytes, you send a XON (0x11) to the sender; the sender will now resume sending data.
  • Go back to (1)

I have never used software serial (does it have a write method?) and I have never used xon/xoff, but you can give it a shot.

You are right I think, I pulled up PUTTY and turned off the XON/XOFF and it was working just fine.

14

Robin2:
When you run into problems the first thing to do is simplify.

Take one of the receive functions out of the program altogether and concentrate on getting the remaining one to work perfectly.

Can you give an example of the data you are trying to receive and an example of the problem that you get.

I don't see why SoftwareSerial should not work properly.

...R

I actually already did this, and it seemed to work fine. So originally, I hooked up a button, that just sent a command to the switch, and it worked fine. My problem was, I didn't have a great way of verifying. I'd have to basically make a "change" that was persistent between me disconnecting the arduino, and reconnecting Putty. It did work though, i was able to change the host name of the switch via the hardware serial. A few have mentioned here that it may be a weakness in the software serial itself, though I'm not sure how I'd know for sure. Sometimes it works correct, most times though there seems to be corrupt data, or extra data that is coming back through.

Most of my issues seem to be communicating responses back from the switch to the arduino.

As far as data goes from the switch, its fairly strait forward. If I were to send a carriage return to it, it would simply respond back with its default prompt. In the case of a cisco switch, that is whatever its hostname is. So it should just say back "LAB-SW-02#" however thats not what happens, it seems to return "LAB-SW-02#" followed directly by some strange text, and then prints the "LAB-SW-02#" again.

I do have a Mega in the mail, but I always just choose the free shipping from China so, it will be here anywhere from 2 weeks to 2 months lol, maybe then I can change over to a hardware serial and see if that makes a difference.

poorSerial2.png
poorSerial2.png765×689 9.05 KB

15

In case anyone is wondering why I am doing this, here is the situation. I work for an IT company, and we support remote customers. Often we will pick up a client who was using a different IT support company before us, and has done a terrible job giving the client the needed settings and configuration of the clients network after switching to us. So we often have remote networks with Firewalls, Routers, and Switches that either have the other IT companies configurations on ( which we'd want to change, at the very least the passwords on ) or it will be network equipment that out of the box works, like Switches, but that really need to be configured remotely.

Because they require a Physical connection to configure correctly, or to turn on remote access services like telnet or ssh on them, we would normally have to hire a secondary company in the area to go out, survey the site, and try and recover the devices for us, and we will have to pay them a premium to do so. Because we have contracts with the client, to provide IT services at X$, these extra tech call outs come right of the top profit wise, and because they do, often are not priorities, or have to be staggered to offset cost. Sometimes we over promise, because we don't know how bad the setup really is.

In comes my solution, for allot of the equipment, we simply need to turn telnet or ssh on. This is not a complex thing to do when you are physically there. So, we take an arduino that can talk with the console port of a device, throw in an SD reader, maybe even a wifi chip that we can configure to auto connect to the clients active AP, and we get a cheap, 15$ console programmer, that we can preload with the configuration we want, or connect to directly via wifi, and program the network device for a fraction of the cost.

Are there things out there like this already? Yes, but not many, and they are more target toward Network Techs that simply want a wireless console port ( no SD pre-configure ) and are wifi hotspots that are connected to with a smart phone or something of that nature.

I don't have the knowledge to do it, but I know it can be done.

Now if I can get the programming to work, it will pretty much be:

  1. Arduino Mini - $2.85
  2. SD Storage Board Module / SPI For Arduino - $0.99
  3. 5V Mini USB Lithium Battery Charging Board - $0.33
  4. 3.7v Cellphone battery - $2.94
  5. DC-DC Step Up Boost Module Converter 3V to 5V - $1.31
  6. MAX3232 MAX3232CSE Serial Port Mini RS232 to TTL - $0.39
  7. ESP8266 Serial WIFI Wireless Transceiver Module - $3.71
    8. Maybe a GUI / OLED screen later.

And I'd have what I think is a solution for under 13$, that could save my company thousands of dollars, and I think my managers would jump on it. Maybe give me a healthy boost on my next review. Plus once it is all working and together, I'll know so much more about arduino, and hopefully the work will be worth something to the community. But getting from that list of parts, to a functional device.

Thats the plan / delusion :smiley:

16

mrm0xie:
I actually already did this, and it seemed to work fine. So originally, I hooked up a button, that just sent a command to the switch, and it worked fine. My problem was, I didn't have a great way of verifying. I'd have to basically make a "change" that was persistent between me disconnecting the arduino, and reconnecting Putty. It did work though, i was able to change the host name of the switch via the hardware serial. A few have mentioned here that it may be a weakness in the software serial itself, though I'm not sure how I'd know for sure. Sometimes it works correct, most times though there seems to be corrupt data, or extra data that is coming back through.

I don't understand how this relates to the business of receiving data. Why is there a switch involved. When I said simplify I really meant it. Write a short program that does nothing to but receive the data and prove that it is received perfectly.

I have no idea what you are trying to describe when you say "persistent between ....." If you disconnect the Arduino it will stop. How can anything persist through that ?

As I said earlier, I would be surprised if there is a problem with SoftwareSerial. Look for a simple problem.

...R

PS, I have now read your Reply#14 and I cannot see how it relates to the problems you have been seeking advice about. I assume you know your business, but you are not making a good job of explaining your problem.