SoftwareSerial magic numbers

I had a look at the magic numbers in SoftwareSerial today (only the 16Mhz part for now) because I wanted to check non-standard serial baudrates too.

static const DELAY_TABLE PROGMEM table[] = 
{
  //  baud    rxcenter   rxintra    rxstop    tx
  { 115200,   1,         17,        17,       12,    },
  { 57600,    10,        37,        37,       33,    },
  { 38400,    25,        57,        57,       54,    },
  { 31250,    31,        70,        70,       68,    },
  { 28800,    34,        77,        77,       74,    },
  { 19200,    54,        117,       117,      114,   },
  { 14400,    74,        156,       156,      153,   },
  { 9600,     114,       236,       236,      233,   },
  { 4800,     233,       474,       474,      471,   },
  { 2400,     471,       950,       950,      947,   },
  { 1200,     947,       1902,      1902,     1899,  },
  { 300,      3804,      7617,      7617,     7614,  },
};

I came up with the following formulas

rxstop = 16000000L/(7 * baudrate) - 2;
rxintra = rxstop;  // an easy one 
tx = rxstop - 4;   // slight error
rxcenter = rxstop/2 - 5;  // slight error at higher baud rates.

this generates the following table

baud	rxcenter	rxintra	rxstop	tx
115200	3	17	17	14
57600	13	37	37	34
38400	23	57	57	54
31250	30	71	71	68
28800	33	77	77	74
19200	53	117	117	114
14400	73	156	156	153
9600	113	236	236	233
4800	232	474	474	471
2400	470	950	950	947
1200	946	1902	1902	1899
300	3803	7617	7617	7614

The difference 'matrix'

2	0	0	2
3	0	0	1
2	0	0	0
1	1	1	0
1	0	0	0
1	0	0	0
1	0	0	0
1	0	0	0
1	0	0	0
1	0	0	0
1	0	0	0
1	0	0	0

Not yet tested thoroughly but the error matrix indicates that the tables can be replaced by formulas, allowing non standard baud rates.

(to be continued)

8 Mhz tables

static const DELAY_TABLE table[] PROGMEM = 
{
  //  baud    rxcenter    rxintra    rxstop  tx
  { 115200,   1,          5,         5,      3,      },
  { 57600,    1,          15,        15,     13,     },
  { 38400,    2,          25,        26,     23,     },
  { 31250,    7,          32,        33,     29,     },
  { 28800,    11,         35,        35,     32,     },
  { 19200,    20,         55,        55,     52,     },
  { 14400,    30,         75,        75,     72,     },
  { 9600,     50,         114,       114,    112,    },
  { 4800,     110,        233,       233,    230,    },
  { 2400,     229,        472,       472,    469,    },
  { 1200,     467,        948,       948,    945,    },
  { 300,      1895,       3805,      3805,   3802,   },
};

the formulas:

int rxstop = 8000000L/(7 * baudrate) - 4;
int rxintra = rxstop;
int tx = rxstop - 3;
int rxcenter = max(rxstop/2 - 7, 1);

results of these formulas

baud	rxcenter	rxintra	rxstop	tx
115200	1	5	5	2
57600	1	15	15	12
38400	5	25	25	22
31250	9	32	32	29
28800	10	35	35	32
19200	20	55	55	52
14400	30	75	75	72
9600	50	115	115	112
4800	110	234	234	231
2400	229	472	472	469
1200	467	948	948	945
300	1895	3805	3805	3802

The difference 'matrix'

0	0	0	1
0	0	0	1
3	0	1	1
2	0	1	0
1	0	0	0
0	0	0	0
0	0	0	0
0	1	1	0
0	1	1	1
0	0	0	0
0	0	0	0
0	0	0	0

I think rxcenter is adjusted a bit to try to account for the ISR overhead. Is that correct? Have you done something similar (is that the "- 5" in the first formulas)?

(is that the "- 5" in the first formulas)

rxcenter is used to get the middle of the pulse, so when reading the bit stream you're not on a falling or rising edge.

What I actually did was place the existing table in Excel; Calculate the "ideal" table (clock/baud) and adjust the numbers for the differences (subtractions).
For the 8Mhz I got a negative value so I adjusted the formula with an additional max() [pragmatic].

First test with Arduino 1.0 SoftwareSerial example code and a Serial LCD - 9600 & 19200 baud - it behaves identical.
Mind you: Both the original table version and the new formula version miss a byte once and a while when sending more than ~10 bytes :frowning:
Adding a delay(1) in the code helps a bit but.....

OK, the changes in SoftwareSerial.cpp are

void SoftwareSerial::begin(long speed)
{
  _rx_delay_centering = _rx_delay_intrabit = _rx_delay_stopbit = _tx_delay = 0;

  // added lines
  long baud = speed;
  _rx_delay_stopbit = 16000000L/(7 * baud) - 2;
  _rx_delay_intrabit = _rx_delay_stopbit;
  _tx_delay = _rx_delay_stopbit - 4;
  _rx_delay_centering = _rx_delay_stopbit/2 - 5;
 
  // 
  // for (unsigned i=0; i<sizeof(table)/sizeof(table[0]); ++i)
  // {
    // long baud = pgm_read_dword(&table[i].baud);
    // if (baud == speed)
    // {
      // _rx_delay_centering = pgm_read_word(&table[i].rx_delay_centering);
      // _rx_delay_intrabit = pgm_read_word(&table[i].rx_delay_intrabit);
      // _rx_delay_stopbit = pgm_read_word(&table[i].rx_delay_stopbit);
      // _tx_delay = pgm_read_word(&table[i].tx_delay);
      // break;
    // }
  // }

  // Set up RX interrupts, but only if we have a valid RX baud rate
  if (_rx_delay_stopbit)
  {
    if (digitalPinToPCICR(_receivePin))
    {
      *digitalPinToPCICR(_receivePin) |= _BV(digitalPinToPCICRbit(_receivePin));
      *digitalPinToPCMSK(_receivePin) |= _BV(digitalPinToPCMSKbit(_receivePin));
    }
    tunedDelay(_tx_delay); // if we were low this establishes the end
  }

#if _DEBUG
  pinMode(_DEBUG_PIN1, OUTPUT);
  pinMode(_DEBUG_PIN2, OUTPUT);
#endif

  listen();
}

The memory size of the 2 sketches:
TABLE BASED : 4484
FORMULA BASED: 4302
==> 182 bytes less

every entry in the DELAY_TABLE = 12 bytes x 12 entries = 144 bytes
so the code is also shorter => no search loop and progmem calls)


in fact as we calculate the baudrate we can also leave out the following lines from SoftwareSerial::begin()

  • _rx_delay_centering = _rx_delay_intrabit = _rx_delay_stopbit = _tx_delay = 0;
  • if (_rx_delay_stopbit)
  • and remove the local variable baud too

difference is now 192 bytes less!

Conclusion:

Not to bad result,

  1. a smaller code base (192 bytes) and
  2. freedom of baud rate selection.

The 2nd is quite useful as now one can adapt the baud rate when the Arduino has no 16.000.000 Hz Crystal but a resonator producing less or more cycles / second.

Pity the SoftwareSerial drops some chars once and a while (original version did that too)

Todo:

  • check every baud rate (old & new)
  • check this code on 1.0.3
  • 20Mhz tables => formulas
  • Due too?
  • test on other Arduino's [MegaTeensy]
  • ...

first version 20 MHz formulas.

static const DELAY_TABLE PROGMEM table[] =
{
  //  baud    rxcenter    rxintra    rxstop  tx
  { 115200,   3,          21,        21,     18,     },
  { 57600,    20,         43,        43,     41,     },
  { 38400,    37,         73,        73,     70,     },
  { 31250,    45,         89,        89,     88,     },
  { 28800,    46,         98,        98,     95,     },
  { 19200,    71,         148,       148,    145,    },
  { 14400,    96,         197,       197,    194,    },
  { 9600,     146,        297,       297,    294,    },
  { 4800,     296,        595,       595,    592,    },
  { 2400,     592,        1189,      1189,   1186,   },
  { 1200,     1187,       2379,      2379,   2376,   },
  { 300,      4759,       9523,      9523,   9520,   },
};

formulas

int rxstop = 20000000L/(7 * baudrate) - 1;
int rxintra = rxstop;
int tx = rxstop - 3;
int rxcenter = rxstop/2 - 4;

115200	7	23	23	20
57600	20	48	48	45
38400	32	73	73	70
31250	41	90	90	87
28800	45	98	98	95
19200	69	147	147	144
14400	94	197	197	194
9600	144	296	296	293
4800	293	594	594	591
2400	590	1189	1189	1186
1200	1185	2379	2379	2376
300	4757	9522	9522	9519

The difference 'matrix'

4	2	2	2
0	5	5	4
5	0	0	0
4	1	1	1
1	0	0	0
2	1	1	1
2	0	0	0
2	1	1	1
3	1	1	1
2	0	0	0
2	0	0	0
2	1	1	1

definitely worse than the 16 and 8 Mhz difference matrices, but it's a start

Still wondering why there is a rxintra and a rxstop column. The values are identical....

@robtillaart,

Noob question, but what's the reason for these values in SoftwareSerial?

Obviously, less memory usage is a good thing. :slight_smile:

submitted as proposal - Proposal to improve SoftwareSerial · Issue #1198 · arduino/Arduino · GitHub -

@Marc G

A serial protocol is like a train with wagons and on each wagon there is one bit, 10 bits in total (including start/stop bits). The baud rate represents the speed of the train.

The software serial receiving code is triggered by the edge of the start bit (train). To read a bit properly one wants to read the value of the signal (HIGH/LOW) in the middle of the bit, not at the edges.

rxCenter is the time to the (approx) middle of the first (start) bit,
rxIntra, rxStop are used as timings from the middle of one bit to the middle of the next. The higher the baud rate the lower these numbers.

tx is used for the timing for the transmit.

You can see this in the code of the library - C:\Program Files (x86)\arduino-1.0\libraries\SoftwareSerial - (windows)
and search for this function - void SoftwareSerial::recv() -

Did a more extensive test with the SoftwareSerial using the formula approach.
I connected two Arduino's - UNO, 16Mhz (resonator) + 2009, 16Mhz (crystal) + IDE 1.0 - both using SoftwareSerial, one Master and the other Slave (essentially an echo).

The master sent byte 0x55 at baudrate 100 and waits until the slave echos it back.
If the answer is not 0x55, the test fails and master prints a message.
Otherwise it just increases the baud rate with 100 and starts over.

The results are pretty good as it only gets constantly distorted above 190K baud.
Between 90K and 190K it only failed 10 times.

I took 0x55 as test pattern 0x55 == 01010101 ; it helps to see what happened. (see comments after output

Typical output (multiple runs had comparable output)
Note: started with baud rate 100 in steps of 100...

start...
BAUD	BYTE
97600	F5	 FAIL  // = 11110101    ??
111200	AA	 FAIL  // = 10101010   1 bit shifted
114400	D5	 FAIL  // = 11010101   1 bit failed (interference with start bit ?
124600	AA	 FAIL
140600	D5	 FAIL
145500	AA	 FAIL
149000	D5	 FAIL
163200	AF	 FAIL  // = 10101111  ??
190500	FF	 FAIL  // = 11111111  expect sync lost 
190600	FF	 FAIL
190700	FF	 FAIL
190800	FF	 FAIL
190900	FF	 FAIL
191000	FF	 FAIL
...

The master and the slave were kept in sync by starting at the same baud rate an wait for each other.

To repeat the test start the master, then start the slave, and press a char in the serial monitor of the master.

Slave program (essentially echo)

//
//    FILE: serialSlave (echo)
//  AUTHOR: Rob Tillaart
//    DATE: 2013-01-02
//
// PUPROSE: test SW serial with formulas
//

#include <SoftwareSerial.h>

SoftwareSerial mySerial(2, 3);

void setup()
{
  Serial.begin(9600);
  Serial.println("start slave...");
}

unsigned long baud=0;

void loop()
{
  baud += 100;
  mySerial.begin(baud);
  while (mySerial.available() == 0);
  int b = mySerial.read();
  mySerial.write(b);
  Serial.println(b,DEC);
  delay(10);
}

master program

//
//    FILE: serialMaster
//  AUTHOR: Rob Tillaart
//    DATE: 2013-01-02
//
// PUPROSE: test SW serial
//

#include <SoftwareSerial.h>

SoftwareSerial mySerial(2, 3);

void setup()
{
  Serial.begin(9600);
  Serial.println("start...");
}

unsigned long baud=0;

void loop()
{
  if (Serial.available() > 0)
  {
    Serial.flush();

    baud += 100;
    mySerial.begin(baud);
    mySerial.write(0x55);
    while (mySerial.available() == 0);
    int b = mySerial.read();
    if (b != 0x55)
    {
      Serial.print(baud);
      Serial.print("\t");
      Serial.print(b, HEX);
      Serial.print("\t");
      Serial.println(" FAIL");
    }
    delay(20);
  }
}

As always comments/remarks are welcome

same test with stepsize 10 gave some more errors (typical run started with baud rate 10, step 10)

start...
BAUD	BYTE
70660	D5	 FAIL
81950	AD	 FAIL
88870	AF	 FAIL
89570	BD	 FAIL
94410	D5	 FAIL
95340	AA	 FAIL
96590	D5	 FAIL
98980	AA	 FAIL
100750	AB	 FAIL
103590	BD	 FAIL
105740	D5	 FAIL
110600	AA	 FAIL
113260	AF	 FAIL
120200	AA	 FAIL

...

Up till 70K no failures ( that are 7000 different baudrates tested !)
between 70K and 115K "only" 13 failures (13 fail on 4500 baudrates tested ~~ 1/300
above 120K the failures increased, not shown

Conclusion from the tests, SoftwareSerial "by formula" works very good up to 70.000 and reasonable well up to 115.200
Tweaking the formulas further may improve the test results but for now I'm quite satisfied.


This SoftwareSerial "by formula" allows one to build a communication channel in which the baud rate is constantly altered, making it very difficult to eavesdrop - and yes to get in sync :slight_smile:

@Rob,

I applaud this effort. Thanks for investigating so thoroughly. I always thought it might be fun to develop some equations that allow the synthesis of the "table" values on the fly, and now it looks like you are pretty close to doing just that.

It's good that you are getting error-free transmission up to 70K. Make sure you test not just the single byte round trip, but also lots of bursts. The values should vary Example:

  1. Arduino sends 0x55 as fast as possible to host for one minute.
  2. Arduino sends 0xFE as fast as possible to host for one minute.
  3. Arduino sends 0x01 as fast as possible to host for one minute.
  4. Host sends 0x55 as fast as possible to Arduino for one minute.
  5. Host sends 0xFE as fast as possible to Arduino for one minute.
  6. Host sends 0x01 as fast as possible to Arduino for one minute.

When constructing the tables, I found several times that I thought the values were good--until I tested the large bursts.

If we want to improve performance at baud rates > 57.6K, I think we're going to have to optimize the timer tick vector. I studied this for some time with the logic analyzer and discovered that the occasional glitch was due to a timer tick interrupt being processed exactly when a pin change was pending.

Lastly, and you probably already know this, but if your formula is off a bit for the lower baud rates, it shouldn't be a big deal. They are very tolerant.

Nice!

Mikal

robtillaart

like the idea of a formula,
but could be slow to change baud rate could it not.

not certain, and for interest,
did you try two boards connected using the standard software serial code,
did you try two boards using the standard hardware uart.

@Dr John,
Yep a changing baudrate communication would certainly be slower than a fixed speed, but calculating the values take micro-seconds, no FP math involved.

I did the test with 2 Arduinos - UNO + DUemillanove - so one with a crystal and one with resonator (?) and used for both SW serial (you could have seen this in the code :wink:

I did not try a HW serial against the SW serial yet although I did test it with a (19200) HW serial LCD - see earlier post.

This analysis is not final yet as I expect the formulas can be improved a bit for the higher speeds. This can be done by non-linear polynomes at the cost of extra footprint or maybe by slighty tuning the constants in the formulas. Need some time to test (a lot more)

@Mikal,
stuff to think through, thanks

well done

testing this sort of thing is a real pain I know,

Tweaked the numbers in the spreadsheet to minimize the cumulative relative error.
There was a large relative error in the higher baud rates, now the relative error is minimized, while keeping the functions linear

(not extensively tested yet)

// 16MHZ
rxstop = 16000000L/(7 * baudrate) - 2;
rxintra = rxstop;  
tx = rxstop - 4; 
rxcenter = rxstop/2 - 7; 

// 8MHZ
rxstop = 8000000L/(7 * baudrate) - 4;
rxintra = rxstop;  
tx = rxstop - 2; 
rxcenter = rxstop/2 - 10; 

// 20MHZ
rxstop = 20000000L/(7 * baudrate) - 3;
rxintra = rxstop;  
tx = rxstop - 3; 
rxcenter = rxstop/2 - 7;

to be continued...

Run with the previous formulas

start...
BAUD	BYTE
70660	D5	 FAIL
81950	AD	 FAIL
88870	AF	 FAIL
89570	BD	 FAIL
94410	D5	 FAIL
95340	AA	 FAIL
96590	D5	 FAIL
98980	AA	 FAIL
100750	AB	 FAIL
103590	BD	 FAIL
105740	D5	 FAIL
110600	AA	 FAIL
113260	AF	 FAIL
120200	AA	 FAIL
...

Now a run with the new offsets

start...
BAUD	BYTE
90440	D5	 FAIL
97150	AD	 FAIL
101140	AA	 FAIL
101210	D5	 FAIL
103180	D5	 FAIL
105430	AA	 FAIL
106130	D5	 FAIL
108400	A9	 FAIL
108990	AA	 FAIL
109440	D5	 FAIL
111270	AA	 FAIL
111320	D5	 FAIL
117300	D5	 FAIL
118480	AA	 FAIL
...

The first fail with new parameters lies about 20 K higher, but other runs started to fail at ~79/80K .

Conclusion for now:
The new offsets are definitely better than the previous, but still not good enough to get a fail free software serial up to 115200.
TODO: test @8Mhz and @20Mhz (don't have such duinos)

A deep dive in the code might be needed.
TBC...