NewSoftSerial Library: An AFSoftSerial update

Hi all--

As an optional companion to TinyGPS, I'm also releasing NewSoftSerial, a library that updates AFSoftSerial in several ways:

  • It corrects a bug in recv() that would sometimes drop received characters
  • It inherits from built-in class Print, eliminating some 4-600 bytes of duplicate code
  • It implements a circular buffering scheme that makes processing RX data more efficient
  • It extends support to all Arduino pins 0-19 (0-21 on Arduino Mini), not just 0-13
  • It supports multiple simultaneous soft serial devices.*
  • But make sure to read the disclaimers in the documentation. :slight_smile:

Mikal

PS Feedback welcome.

M

Awesome. I'll look into making this the new default SoftwareSerial library for Arduino.

I'm having some trouble with the NewSoftSerial library. Serial receives are hanging the program. It seems like single characters received very slowly will work fine. But as soon as multiple characters are received quickly, the program freezes. Sending seems to work without any problems.

Here's some sample code. It's basically a baud rate converter. The hardware UART is configured for 9600 baud and the NewSoftSerial instance is set for 2400 baud (I've tried other speeds and it made no difference on the locking). Any data received by the hardware UART is sent out through NewSoftSerial and vice versa.

#include <NewSoftSerial.h>
#define rxPin 2
#define txPin 3
#define ledPin 13

boolean ledState = false;
int counter = 255;            // To slow down the LED blinking
byte incomingByte = 0;

NewSoftSerial nss(rxPin, txPin);

void setup()                    // run once, when the sketch starts
{
  Serial.begin(9600);
  nss.begin(2400);
  pinMode(ledPin, OUTPUT);      // sets the digital pin as output
  digitalWrite(ledPin,ledState);
  delay(2000);
}

void loop()                     // run over and over again
{
  
// Blink the LED on pin 13 just to show we're alive

  if (counter > 0) {
    counter-=1;
  } else {
    ledState = ! ledState;
    digitalWrite(ledPin,ledState);
    counter=255;
  }

// Read from the hardware UART.
// If any data is available, write it out through the software serial

  if (Serial.available() > 0) {
    incomingByte = Serial.read();
    nss.print(incomingByte);
  }

// Read from software serial.
// If any data is available, write it out through the hardware UART
  
  if (nss.available() > 0) {
    incomingByte = nss.read();
    Serial.print(incomingByte);
  }

  delay(1);                  
}

I thought there might be an interrupt conflict between the hardware UART and the NewSoftSerial but commenting out the hardware Serial code doesn't prevent lockups. Even this simplified code will freeze after a few characters are received rapidly by NewSoftSerial.

#include <NewSoftSerial.h>
#define rxPin 2
#define txPin 3
#define ledPin 13

boolean ledState = false;
int counter = 255;            // To slow down the LED blinking

NewSoftSerial nss(rxPin, txPin);

void setup()                    // run once, when the sketch starts
{
  nss.begin(2400);
  pinMode(ledPin, OUTPUT);      // sets the digital pin as output
  digitalWrite(ledPin,ledState);
  delay(2000);
}

void loop()                     // run over and over again
{
  
// Blink the LED on pin 13 just to show we're alive

  if (counter > 0) {
    counter-=1;
  } else {
    ledState = ! ledState;
    digitalWrite(ledPin,ledState);
    counter=255;
  }
  
  delay(1);                  
}

Notice that I'm not even calling any NewSoftSerial functions (except for the begin) or even using the hardware Serial at all. Actually the program isn't doing anything but initializing the NewSoftSerial instance and blinking a LED.

Hopefully this is enough info to help find the problem as I'd really like to use this library.

I forgot to add that this is with Arduino 0012 running on a G5 (PPC) Mac. I downloaded the latest NewSoftSerial library today.

etracer--

Thanks very much for testing out NewSoftSerial. I tried your second example with both 0012 and the unreleased 0013 and do not see the lockup you are experiencing. I do notice that when I connect my GPS to the software serial port that the rate of flicker in the LED becomes slightly uneven, but I guess I would expect this. We are getting hundreds of interrupts per second after all.

What device is connected to your software serial port?

As a test, would you please replace NewSoftSerial with AFSoftSerial? Since NewSoftSerial inherits much of its interrupt code from AFSoftSerial, this will be a very interesting test. It should be an exact replacement in this case.

In your baud rate converter -- excellent project by the way -- you need to change

nss.print(incomingByte);

to

nss.print(incomingByte, BYTE);

and similarly with the Serial.print.

EDIT: I just tried your baud rate converter. Running the NewSoftSerial at 4800 and the Serial console at 9600, I get my GPS data in the console beautifully. If I ratchet Serial and the console down to 2400 I still get valid data, but some of it is lost -- exactly as you'd expect.

Mikal

I've tried a variety of devices with the software serial. Originally I started testing with connecting to a APC SmartUPS at 2400 baud through a MAX232. When I was getting the unexplained lockups, I connected next to a PC serial port (again through the MAX232) so I could manually send data to the software serial. Trying to eliminate the MAX232 as the possible problem, I next connected directly to a ID-12 RFID module which outputs TTL-level serial (at 9600 baud). In all cases the results were the same and the software serial locks up after a few rapid characters. It's random as to how many characters make it before the lock.

I tried AFSoftSerial and get the same lockups.

I did some digging in the source and was able to confirm the problem is somewhere in the recv() called by the interrupts. Commenting out the code (but leaving the definition) in NewSoftSerial::recv() stops the lockups - but obviously also stops any data. In playing with the code I find that if I leave any more code then the initial digitalRead if, I get lockups. So this code doesn't lock up (but also obviously doesn't return any data).

void NewSoftSerial::recv() 
{

  char i, d = 0; 

  if (digitalRead(_receivePin)) 
    return;       // not ready! 
/*

  tunedDelay(_bitDelay - 8);

  for (i=0; i<8; i++) { 
    //PORTB |= _BV(5); 
    tunedDelay(_bitDelay*2 - 6);  // digitalread takes some time
    //PORTB &= ~_BV(5); 
    if (digitalRead(_receivePin)) 
      d |= (1 << i); 
  } 
  tunedDelay(_bitDelay*2);

  // buffer full?
  if ((_receive_buffer_tail + 1) % _NewSS_MAX_RX_BUFF == _receive_buffer_head)
    return;

  _receive_buffer[_receive_buffer_tail++] = d; // save new byte 
  if (_receive_buffer_tail == _NewSS_MAX_RX_BUFF)
    _receive_buffer_tail = 0;

*/
}

I think the problem might either have something to do with calling the digitalRead function or simply having the code running during the interrupt taking too long. As a trivial case, I replaced the recv() function with:

void NewSoftSerial::recv() 
{

  if (digitalRead(_receivePin)) 
    return;       // not ready! 
}

While the code will obviously return no data, I also get no lockups. If I add one more digitalRead() like this:

void NewSoftSerial::recv() 
{

  if (digitalRead(_receivePin)) 
    return;       // not ready! 

  if (digitalRead(_receivePin)) 
    return;       // not ready! 
}

Then I get the lockups.

Another probably unrelated weirdness that I stumbled across is with the tunedDelay() assembly. While commenting out different bits of code to test, I found that if I had two calls to tunedDelay() one after another, I would get the following compile errors:

/var/tmp//cc0RgyXU.s: Assembler messages:
/var/tmp//cc0RgyXU.s:940: Error: register r24, r26, r28 or r30 required

I don't know AVR assembly so I can't really debug further. Maybe there's a word alignment problem when the two assembly blocks are placed one after another?

I can't explain why your tests work fine and every test I try fails with lockups. I've tried an Arduino NG, an Adafruit Boardurino, and a bare ATmega 168 on a breadboard with the Diecimila bootloader and get the same results with each (all at 16MHz). I've even tried different pins for the software serial with no change.

Hopefully this info helps.

Thanks

That sounds like some very thorough testing, but alas, doesn't seem to get me any closer to an understanding. I want to say that there is something about your configuration that is causing the problem, but as you have tested many different serial devices with many different Arduino flavors, I don't know. All I can say is that a lot of people use AFSoftSerial and I can't remember anyone reporting this kind of behavior. There are reports of baud rates not working quite right, but I am investigating some of these.

Are you using a special bootloader? Are you using Ubuntu by any chance?

Questions for the Arduino world at large:

  1. Has anyone experienced the kind of lockup in AFSoftSerial that etracer describes?
  2. Please let me know if you
    a. notice any difference in behavior between AFSoftSerial and NewSoftSerial.
    b. have successfully tested multiple instances of NewSoftSerial.
    c. have any observations about baud rates that don't seem to work reliably with either AFSoftSerial or NewSoftSerial.

Mikal

mikalhart: what OS are you on? I've heard reports that some versions of avr-gcc have problems with functions called by interrupts. Which versions of avr-gcc do you two have?

I've tried a variety of bootloaders as well. The Arduino NG has the standard bootloader. The Boardurino has a slightly modified "no-wait" bootloader by LadyAda. The bare ATmega168 was burned directly from a .hex file and had no bootloader.

I'm using a G5 Mac running OSX 10.4.10, but that's probably not relevant the lockups happen even if the Arduino is not connected to USB and running off a separate power supply. I suppose there's always a possibility of build corruption or differences. To eliminate this I just took the code over to an Intel machine running OSX 10.5.6 and rebuilt and re-uploaded the program to the Arduino NG. No difference. Still get the lockups.

mellis: I'm using 0012 and 0013 exclusively on Windows. [EDIT: irrelevant stuff deleted.]

The compiler that comes with Windows 0013 is avr-gcc 4.3.0, which is the one that the avr-libc developers advised steering clear of. But I speculate that the reason I and other Windows users don't see problems like the ones above is that we don't build our own avr-libc -- we use the one in the distro.

Whether any of this I've just typed is related to etracer's issues with AFSoftSerial and NewSoftSerial is just speculation.

Mikal

mellis:

I'm running the included version in Arduino 0012 for OSX. The versions.txt file contains:

avarice: 2.7
avr-libc: 1.6.2
avrdude: 5.5
binutils: 2.18
gcc-3: 3.4.6
gcc-4: 4.3.0
gdb: 6.6
libusb: 0.1.12
make: 3.81
simulavr: 0.1.2.5

Running avr-gcc -v reports:

$ ./avr-gcc -v
Using built-in specs.
Target: avr
Configured with: ../configure --prefix=/usr/local/AVRMacPack-20080514 --disable-dependency-tracking --disable-nls --target=avr --enable-languages=c,c++ --disable-nls --disable-libssp --with-dwarf2
Thread model: single
gcc version 4.3.0 (GCC)

mellis:

Looks like you nailed it!

I installed Arduino 0012 on a Windows XP machine and rebuilt and uploaded my test program. Everything works correctly with no lockups.

So now the question is: how do I resolve this avr-gcc version/interrupt problem on OSX? Is there a later/different version in the Arduino 0013 build?

Questions for the Arduino world at large:

  1. Has anyone experienced the kind of lockup in AFSoftSerial that etracer describes?
  2. Please let me know if you
    a. notice any difference in behavior between AFSoftSerial and NewSoftSerial.
    b. have successfully tested multiple instances of NewSoftSerial.
    c. have any observations about baud rates that don't seem to work reliably with either AFSoftSerial or NewSoftSerial.

Mikal

I have tested both AFSoftSerial.h and NewSoftSerial.h with 2400 baud rate for a Parallax RFID reader. Both produce lockups - although they seem to lock at slightly different points in the code. The exact same code works perfectly with SoftwareSerial.h

I am using OSX 10.4.11 with Arduino 0012.

rogermm

Sigh. It does seem like something is broken in OSX interrupts.

I've been playing around with NewSoftSerial and it's been working pretty well for me, with only a few difficulties so far. Thanks for your great work packaging up a bunch of different ideas, mikalhart.

A few notes:

  • I've experienced some problems doing the same sort of serial pass-through, where you just echo characters between the softserial pins and the UART. If the device I'm speaking to on the softserial port is set to echo, no matter how slowly I type I am liable to get garbage characters interspersed with what I'm typing. Turning off echo and having everything run half duplex gives pretty good results. Could this be because pins 2 and 3 (used very often in the soft serial examples I've seen) share an interrupt with the UART on pins 0 and 1? Unfortunately, my device requires a specially designed shield so I can't move pin assignments around easily, but if someone who is having interrupt problems could try using, say, pins 8 and 9 for the soft serial and see if they continue to have problems, that would be an interesting experiment.

  • Another problem I've had is when I get a whole slew of characters sent back from the serial device, it quickly fills the buffer and I lose the rest of the data, even when I'm not doing hardly anything else in the main loop other than trying to handle the input. I'm looking into adding proper RTS/CTS handling as an option to NewSoftSerial to see if that helps. There's an attempt at CTS handling in the AF_XPort code, but it doesn't seem to work properly for me.

Could this be because pins 2 and 3 (used very often in the soft serial examples I've seen) share an interrupt with the UART on pins 0 and 1? Unfortunately, my device requires a specially designed shield so I can't move pin assignments around easily, but if someone who is having interrupt problems could try using, say, pins 8 and 9 for the soft serial and see if they continue to have problems, that would be an interesting experiment.

I was using pins 2 and 3 and have changed them to 8 and 9 as suggested - unfortunately there is no noticeable change in the lockup.

Really hope this can be resolved.. :slight_smile:

I have posted an update to NewSoftSerial (http://www.sundial.org/arduino/?page_id=61). The new version adds support for 300, 1200, 14400, and 28800 baud, although you should use caution with 300 and 1200 as the long interrupt times causes interrupt starvation and the millis() clock doesn't work correctly.

There is also a new self clearing overflow flag, so that you can see if you have lost any characters due to buffer overflow:

if (nss.overflow())
  Serial.print("Characters lost!");

The biggest improvement is in the tuning of higher baud rates. With this new version, I get 100% reliability on 38.4K, 57.6K, and 115.2K transmits, 99.9% on 38.4K receives and 99.7% on 57.6K receives. (115.2K is too fast for RX.) I'd love to see some testing with other high speed serial devices.

@rogermm: I think the prevailing theory is that there is something wrong with the OSX build process for interrupt type applications. I'd like to prove this though. Perhaps if you post or PM me your code, I can take a look at it, rebuild it here, and send you the binary to upload.

@jin: Would you please try AFSoftSerial as a control? At this point I am most interested in identifying and fixing any problems that I introduced with NewSoftSerial. Later I think we can attack anything that is latent in AFSoftSerial.

Thanks for the comments.

Mikal

The problem is definitely with avr-gcc 4.3 on OSX. Building the same source on Windows with Arduino 0012 or 0013 works just fine with no lockups.

Here's a work-around: Take the object file for the library from a Windows machine and copy it to the NewSoftWerial library folder on OSX. Since the NewSoftSerial.o file is already there, the code won't be recompiled. I tested this on my OSX machine and was able to compile my program on OSX and upload to the Arduino and everything ran perfectly.

mikalhart: Maybe you can build a separate download for OSX users that already has a NewSoftSerial.o file from Windows? Not an ideal solution by any means, but it's better then nothing. Also include a disclaimer as to what target chip the code was built for as a different target would need a recompile (and then it wouldn't work anymore).

If I'm not mistaken, object code is compiled for a specific microcontroller, and is recompiled if you select a board type that uses a different processor. This means your object code distribution won't work (more precisely, it would only work for one ATmega and would be broken if the user switched board types).

-j

Hence the reason I said:

Also include a disclaimer as to what target chip the code was built for as a different target would need a recompile (and then it wouldn't work anymore).

Hi Mikal

my code is for a wireless RFID reader. I would like to check battery levels and look for tags in my main loop. As has been reported in other threads - http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1201703850/9#9 - SoftwareSerial's read function blocks until there is a tag available to read, making it 'impossible' to perform both checks. The available function in NewSoftSerial would hopefully resolve this.

Much thanks for the support.
Roger :slight_smile:

#include <NewSoftSerial.h>
#include <avr/sleep.h>
#include <string.h>

#define BATTERY_CHECK_INTERVAL 60000
#define SIX_HOUR 952
#define FIVE_HOUR 945
#define FOUR_HOUR 936
#define THREE_HOUR 926
#define TWO_HOUR 910
#define ONE_HOUR 881
#define ZERO_HOUR 836


// RFID reader for Arduino
// Wiring version by BARRAGAN <http://people.interaction-ivrea.it/h.barragan>
// Modified for Arudino by djmatic
// Modified to use SoftwareSerial by Erik Sjodin <http://www.eriksjodin.net>
// Modified for use in TagMat by Roger Meintjes <http://www.doc.gold.ac.uk/~ma701rm/>
 
/*
 * RFID_Scratch03 (Arduino)
 * 
 * Tests communication between RFID, ArduinoXbee, Processing & Scratch.
 * Reads in RFID tag code from reader and writes it out through Xbee.
 * Code is written out as the String start_byte<10 char code>stop_byte.
 *
 * Battery is checked once every minute.
 * Battery charge written out as String start_byte<10 char charge in hours>stop_byte.
 *
 * Connect:
 * Arduino pin 3 to RFID SOUT
 * Arduino GND to RFID GND
 * Arduino Digital pin 2 to RFID /ENABLE
 * Arduino 5v to RFID VCC
 */

 
char code[13] = "";  
char charge[13] = "";
char baseString[11] = {72, 82, 95, 67, 72, 65, 82, 71, 69, 13, 0};   

unsigned long timeNow = 0;
int batteryLevel = 0;
int val = 0;
int bytesRead = 0;

int rx = 3;
int tx = 4;
int batteryPin = 5;
int rfidEnable = 2;

//set up software serial port
NewSoftSerial softSerial = NewSoftSerial(rx, tx);

void setup()
{
  timeNow = millis();
  pinMode(rx, INPUT);
  pinMode(tx, OUTPUT);
  pinMode(batteryPin, INPUT);
  pinMode(rfidEnable, OUTPUT); 
  
  softSerial.begin(2400);     //set software serial port to reader baud rate
  Serial.begin(9600);         //set hardware serial port to xbee baud rate
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);   //set sleep_mode to most power saving   
  
  //initialize reader with reset cycle
  digitalWrite(rfidEnable, LOW);       
  delay(100);
  digitalWrite(rfidEnable, HIGH);
  delay(100);
  digitalWrite(rfidEnable, LOW);
  delay(100);
}

void loop() {
  if((millis() - timeNow) > BATTERY_CHECK_INTERVAL) {
    checkBattery();
    timeNow = millis();
  }
  else {
    checkReader();
  }
}

void checkReader() {
  digitalWrite(rfidEnable, LOW);
  if(softSerial.available() > 0) {
    //check for start byte
    if((val = softSerial.read()) == 10) {
      bytesRead = 0;
      code[bytesRead] = val;
      bytesRead++;
      //do nothing until all twelve bytes have arrived
      while(softSerial.available() < 12);
      //read in code and stop byte
      while(bytesRead < 12) {
        val = softSerial.read();
        code[bytesRead] = val;
        bytesRead++;
      }
      digitalWrite(rfidEnable, HIGH);
      //check for the stop byte
      if(val == 13) {
        Serial.print(code);
      }
      delay(250);      
    }
  }
}

void checkBattery() {
  batteryLevel = analogRead(batteryPin);
  charge[0] = 10;  //start byte
  charge[2] = 0;   //termination byte
  if(batteryLevel > SIX_HOUR) {
    charge[1] = 54;   //ASCII 6
  }
  else if(batteryLevel > FIVE_HOUR) {
    charge[1] = 53;   //ASCII 5
  }
  else if(batteryLevel > FOUR_HOUR) {
    charge[1] = 52;   //ASCII 4
  }
  else if(batteryLevel > THREE_HOUR) {
    charge[1] = 51;   //ASCII 3
  }
  else if(batteryLevel > TWO_HOUR) {
    charge[1] = 50;   //ASCII 2
  }
  else if(batteryLevel > ONE_HOUR) {
    charge[1] = 49;   //ASCII 1
  }
  else if(batteryLevel > ZERO_HOUR) {
    charge[1] = 48;   //ASCII 0
  }
  else {
    sleep_mode();
  }
  Serial.print(strcat(charge, baseString));  //create and write charge string
}