SoftwareSerial magic numbers

I had a look at the magic numbers in SoftwareSerial today (only the 16Mhz part for now) because I wanted to check non-standard serial baudrates too.

static const DELAY_TABLE PROGMEM table[] = 
{
  //  baud    rxcenter   rxintra    rxstop    tx
  { 115200,   1,         17,        17,       12,    },
  { 57600,    10,        37,        37,       33,    },
  { 38400,    25,        57,        57,       54,    },
  { 31250,    31,        70,        70,       68,    },
  { 28800,    34,        77,        77,       74,    },
  { 19200,    54,        117,       117,      114,   },
  { 14400,    74,        156,       156,      153,   },
  { 9600,     114,       236,       236,      233,   },
  { 4800,     233,       474,       474,      471,   },
  { 2400,     471,       950,       950,      947,   },
  { 1200,     947,       1902,      1902,     1899,  },
  { 300,      3804,      7617,      7617,     7614,  },
};

I came up with the following formulas

rxstop = 16000000L/(7 * baudrate) - 2;
rxintra = rxstop;  // an easy one 
tx = rxstop - 4;   // slight error
rxcenter = rxstop/2 - 5;  // slight error at higher baud rates.

this generates the following table

baud    rxcenter    rxintra rxstop  tx
115200  3   17  17  14
57600   13  37  37  34
38400   23  57  57  54
31250   30  71  71  68
28800   33  77  77  74
19200   53  117 117 114
14400   73  156 156 153
9600    113 236 236 233
4800    232 474 474 471
2400    470 950 950 947
1200    946 1902    1902    1899
300 3803    7617    7617    7614

The difference 'matrix'

2   0   0   2
3   0   0   1
2   0   0   0
1   1   1   0
1   0   0   0
1   0   0   0
1   0   0   0
1   0   0   0
1   0   0   0
1   0   0   0
1   0   0   0
1   0   0   0

Not yet tested thoroughly but the error matrix indicates that the tables can be replaced by formulas, allowing non standard baud rates.

(to be continued)

8 Mhz tables

static const DELAY_TABLE table[] PROGMEM = 
{
  //  baud    rxcenter    rxintra    rxstop  tx
  { 115200,   1,          5,         5,      3,      },
  { 57600,    1,          15,        15,     13,     },
  { 38400,    2,          25,        26,     23,     },
  { 31250,    7,          32,        33,     29,     },
  { 28800,    11,         35,        35,     32,     },
  { 19200,    20,         55,        55,     52,     },
  { 14400,    30,         75,        75,     72,     },
  { 9600,     50,         114,       114,    112,    },
  { 4800,     110,        233,       233,    230,    },
  { 2400,     229,        472,       472,    469,    },
  { 1200,     467,        948,       948,    945,    },
  { 300,      1895,       3805,      3805,   3802,   },
};

the formulas:

int rxstop = 8000000L/(7 * baudrate) - 4;
int rxintra = rxstop;
int tx = rxstop - 3;
int rxcenter = max(rxstop/2 - 7, 1);

results of these formulas

baud    rxcenter    rxintra rxstop  tx
115200  1   5   5   2
57600   1   15  15  12
38400   5   25  25  22
31250   9   32  32  29
28800   10  35  35  32
19200   20  55  55  52
14400   30  75  75  72
9600    50  115 115 112
4800    110 234 234 231
2400    229 472 472 469
1200    467 948 948 945
300 1895    3805    3805    3802

The difference 'matrix'

0   0   0   1
0   0   0   1
3   0   1   1
2   0   1   0
1   0   0   0
0   0   0   0
0   0   0   0
0   1   1   0
0   1   1   1
0   0   0   0
0   0   0   0
0   0   0   0

I think rxcenter is adjusted a bit to try to account for the ISR overhead. Is that correct? Have you done something similar (is that the "- 5" in the first formulas)?

(is that the "- 5" in the first formulas)

rxcenter is used to get the middle of the pulse, so when reading the bit stream you're not on a falling or rising edge.

What I actually did was place the existing table in Excel; Calculate the "ideal" table (clock/baud) and adjust the numbers for the differences (subtractions). For the 8Mhz I got a negative value so I adjusted the formula with an additional max() [pragmatic].

First test with Arduino 1.0 SoftwareSerial example code and a Serial LCD - 9600 & 19200 baud - it behaves identical. Mind you: Both the original table version and the new formula version miss a byte once and a while when sending more than ~10 bytes :( Adding a delay(1) in the code helps a bit but.....

OK, the changes in SoftwareSerial.cpp are

void SoftwareSerial::begin(long speed)
{
  _rx_delay_centering = _rx_delay_intrabit = _rx_delay_stopbit = _tx_delay = 0;

  // added lines
  long baud = speed;
  _rx_delay_stopbit = 16000000L/(7 * baud) - 2;
  _rx_delay_intrabit = _rx_delay_stopbit;
  _tx_delay = _rx_delay_stopbit - 4;
  _rx_delay_centering = _rx_delay_stopbit/2 - 5;

  // 
  // for (unsigned i=0; i

The memory size of the 2 sketches: TABLE BASED : 4484 FORMULA BASED: 4302 ==> 182 bytes less

every entry in the DELAY_TABLE = 12 bytes x 12 entries = 144 bytes so the code is also shorter => no search loop and progmem calls)


in fact as we calculate the baudrate we can also leave out the following lines from SoftwareSerial::begin() - _rx_delay_centering = _rx_delay_intrabit = _rx_delay_stopbit = _tx_delay = 0; - if (_rx_delay_stopbit) - and remove the local variable baud too

difference is now 192 bytes less!

Conclusion:

Not to bad result, 1) a smaller code base (192 bytes) and 2) freedom of baud rate selection.

The 2nd is quite useful as now one can adapt the baud rate when the Arduino has no 16.000.000 Hz Crystal but a resonator producing less or more cycles / second.

Pity the SoftwareSerial drops some chars once and a while (original version did that too)

Todo: - check every baud rate (old & new) - check this code on 1.0.3 - 20Mhz tables => formulas - Due too? - test on other Arduino's [MegaTeensy] - ...

first version 20 MHz formulas.

static const DELAY_TABLE PROGMEM table[] =
{
  //  baud    rxcenter    rxintra    rxstop  tx
  { 115200,   3,          21,        21,     18,     },
  { 57600,    20,         43,        43,     41,     },
  { 38400,    37,         73,        73,     70,     },
  { 31250,    45,         89,        89,     88,     },
  { 28800,    46,         98,        98,     95,     },
  { 19200,    71,         148,       148,    145,    },
  { 14400,    96,         197,       197,    194,    },
  { 9600,     146,        297,       297,    294,    },
  { 4800,     296,        595,       595,    592,    },
  { 2400,     592,        1189,      1189,   1186,   },
  { 1200,     1187,       2379,      2379,   2376,   },
  { 300,      4759,       9523,      9523,   9520,   },
};

formulas

int rxstop = 20000000L/(7 * baudrate) - 1;
int rxintra = rxstop;
int tx = rxstop - 3;
int rxcenter = rxstop/2 - 4;

115200  7   23  23  20
57600   20  48  48  45
38400   32  73  73  70
31250   41  90  90  87
28800   45  98  98  95
19200   69  147 147 144
14400   94  197 197 194
9600    144 296 296 293
4800    293 594 594 591
2400    590 1189    1189    1186
1200    1185    2379    2379    2376
300 4757    9522    9522    9519

The difference 'matrix'

4   2   2   2
0   5   5   4
5   0   0   0
4   1   1   1
1   0   0   0
2   1   1   1
2   0   0   0
2   1   1   1
3   1   1   1
2   0   0   0
2   0   0   0
2   1   1   1

definitely worse than the 16 and 8 Mhz difference matrices, but it's a start

Still wondering why there is a rxintra and a rxstop column. The values are identical....

@robtillaart,

Noob question, but what's the reason for these values in SoftwareSerial?

Obviously, less memory usage is a good thing. :)

submitted as proposal - https://github.com/arduino/Arduino/issues/1198 -

@Marc G

A serial protocol is like a train with wagons and on each wagon there is one bit, 10 bits in total (including start/stop bits). The baud rate represents the speed of the train.

The software serial receiving code is triggered by the edge of the start bit (train). To read a bit properly one wants to read the value of the signal (HIGH/LOW) in the middle of the bit, not at the edges.

rxCenter is the time to the (approx) middle of the first (start) bit, rxIntra, rxStop are used as timings from the middle of one bit to the middle of the next. The higher the baud rate the lower these numbers.

tx is used for the timing for the transmit.

You can see this in the code of the library - C:\Program Files (x86)\arduino-1.0\libraries\SoftwareSerial - (windows) and search for this function - void SoftwareSerial::recv() -

Did a more extensive test with the SoftwareSerial using the formula approach. I connected two Arduino's - UNO, 16Mhz (resonator) + 2009, 16Mhz (crystal) + IDE 1.0 - both using SoftwareSerial, one Master and the other Slave (essentially an echo).

The master sent byte 0x55 at baudrate 100 and waits until the slave echos it back. If the answer is not 0x55, the test fails and master prints a message. Otherwise it just increases the baud rate with 100 and starts over.

The results are pretty good as it only gets constantly distorted above 190K baud. Between 90K and 190K it only failed 10 times.

I took 0x55 as test pattern 0x55 == 01010101 ; it helps to see what happened. (see comments after output

Typical output (multiple runs had comparable output) Note: started with baud rate 100 in steps of 100...

start...
BAUD    BYTE
97600   F5   FAIL  // = 11110101    ??
111200  AA   FAIL  // = 10101010   1 bit shifted
114400  D5   FAIL  // = 11010101   1 bit failed (interference with start bit ?
124600  AA   FAIL
140600  D5   FAIL
145500  AA   FAIL
149000  D5   FAIL
163200  AF   FAIL  // = 10101111  ??
190500  FF   FAIL  // = 11111111  expect sync lost 
190600  FF   FAIL
190700  FF   FAIL
190800  FF   FAIL
190900  FF   FAIL
191000  FF   FAIL
...

The master and the slave were kept in sync by starting at the same baud rate an wait for each other.

To repeat the test start the master, then start the slave, and press a char in the serial monitor of the master.

Slave program (essentially echo)

//
//    FILE: serialSlave (echo)
//  AUTHOR: Rob Tillaart
//    DATE: 2013-01-02
//
// PUPROSE: test SW serial with formulas
//

#include 

SoftwareSerial mySerial(2, 3);

void setup()
{
  Serial.begin(9600);
  Serial.println("start slave...");
}

unsigned long baud=0;

void loop()
{
  baud += 100;
  mySerial.begin(baud);
  while (mySerial.available() == 0);
  int b = mySerial.read();
  mySerial.write(b);
  Serial.println(b,DEC);
  delay(10);
}

master program

//
//    FILE: serialMaster
//  AUTHOR: Rob Tillaart
//    DATE: 2013-01-02
//
// PUPROSE: test SW serial
//

#include 

SoftwareSerial mySerial(2, 3);

void setup()
{
  Serial.begin(9600);
  Serial.println("start...");
}

unsigned long baud=0;

void loop()
{
  if (Serial.available() > 0)
  {
    Serial.flush();

    baud += 100;
    mySerial.begin(baud);
    mySerial.write(0x55);
    while (mySerial.available() == 0);
    int b = mySerial.read();
    if (b != 0x55)
    {
      Serial.print(baud);
      Serial.print("\t");
      Serial.print(b, HEX);
      Serial.print("\t");
      Serial.println(" FAIL");
    }
    delay(20);
  }
}

As always comments/remarks are welcome

same test with stepsize 10 gave some more errors (typical run started with baud rate 10, step 10)

start...
BAUD    BYTE
70660   D5   FAIL
81950   AD   FAIL
88870   AF   FAIL
89570   BD   FAIL
94410   D5   FAIL
95340   AA   FAIL
96590   D5   FAIL
98980   AA   FAIL
100750  AB   FAIL
103590  BD   FAIL
105740  D5   FAIL
110600  AA   FAIL
113260  AF   FAIL
120200  AA   FAIL

...

Up till 70K no failures ( that are 7000 different baudrates tested !) between 70K and 115K "only" 13 failures (13 fail on 4500 baudrates tested ~~ 1/300 above 120K the failures increased, not shown

Conclusion from the tests, SoftwareSerial "by formula" works very good up to 70.000 and reasonable well up to 115.200 Tweaking the formulas further may improve the test results but for now I'm quite satisfied.


This SoftwareSerial "by formula" allows one to build a communication channel in which the baud rate is constantly altered, making it very difficult to eavesdrop - and yes to get in sync :)

@Rob,

I applaud this effort. Thanks for investigating so thoroughly. I always thought it might be fun to develop some equations that allow the synthesis of the "table" values on the fly, and now it looks like you are pretty close to doing just that.

It's good that you are getting error-free transmission up to 70K. Make sure you test not just the single byte round trip, but also lots of bursts. The values should vary Example:

  1. Arduino sends 0x55 as fast as possible to host for one minute.
  2. Arduino sends 0xFE as fast as possible to host for one minute.
  3. Arduino sends 0x01 as fast as possible to host for one minute.
  4. Host sends 0x55 as fast as possible to Arduino for one minute.
  5. Host sends 0xFE as fast as possible to Arduino for one minute.
  6. Host sends 0x01 as fast as possible to Arduino for one minute.

When constructing the tables, I found several times that I thought the values were good--until I tested the large bursts.

If we want to improve performance at baud rates > 57.6K, I think we're going to have to optimize the timer tick vector. I studied this for some time with the logic analyzer and discovered that the occasional glitch was due to a timer tick interrupt being processed exactly when a pin change was pending.

Lastly, and you probably already know this, but if your formula is off a bit for the lower baud rates, it shouldn't be a big deal. They are very tolerant.

Nice!

Mikal

robtillaart

like the idea of a formula, but could be slow to change baud rate could it not.

not certain, and for interest, did you try two boards connected using the standard software serial code, did you try two boards using the standard hardware uart.

@Dr John,
Yep a changing baudrate communication would certainly be slower than a fixed speed, but calculating the values take micro-seconds, no FP math involved.

I did the test with 2 Arduinos - UNO + DUemillanove - so one with a crystal and one with resonator (?) and used for both SW serial (you could have seen this in the code :wink:

I did not try a HW serial against the SW serial yet although I did test it with a (19200) HW serial LCD - see earlier post.

This analysis is not final yet as I expect the formulas can be improved a bit for the higher speeds. This can be done by non-linear polynomes at the cost of extra footprint or maybe by slighty tuning the constants in the formulas. Need some time to test (a lot more)

@Mikal, stuff to think through, thanks

well done

testing this sort of thing is a real pain I know,

Tweaked the numbers in the spreadsheet to minimize the cumulative relative error. There was a large relative error in the higher baud rates, now the relative error is minimized, while keeping the functions linear

(not extensively tested yet)

// 16MHZ
rxstop = 16000000L/(7 * baudrate) - 2;
rxintra = rxstop;  
tx = rxstop - 4; 
rxcenter = rxstop/2 - 7; 

// 8MHZ
rxstop = 8000000L/(7 * baudrate) - 4;
rxintra = rxstop;  
tx = rxstop - 2; 
rxcenter = rxstop/2 - 10; 

// 20MHZ
rxstop = 20000000L/(7 * baudrate) - 3;
rxintra = rxstop;  
tx = rxstop - 3; 
rxcenter = rxstop/2 - 7;

to be continued...

Run with the previous formulas

start...
BAUD    BYTE
70660   D5   FAIL
81950   AD   FAIL
88870   AF   FAIL
89570   BD   FAIL
94410   D5   FAIL
95340   AA   FAIL
96590   D5   FAIL
98980   AA   FAIL
100750  AB   FAIL
103590  BD   FAIL
105740  D5   FAIL
110600  AA   FAIL
113260  AF   FAIL
120200  AA   FAIL
...

Now a run with the new offsets

start...
BAUD    BYTE
90440   D5   FAIL
97150   AD   FAIL
101140  AA   FAIL
101210  D5   FAIL
103180  D5   FAIL
105430  AA   FAIL
106130  D5   FAIL
108400  A9   FAIL
108990  AA   FAIL
109440  D5   FAIL
111270  AA   FAIL
111320  D5   FAIL
117300  D5   FAIL
118480  AA   FAIL
...

The first fail with new parameters lies about 20 K higher, but other runs started to fail at ~79/80K .

Conclusion for now: The new offsets are definitely better than the previous, but still not good enough to get a fail free software serial up to 115200. TODO: test @8Mhz and @20Mhz (don't have such duinos)

A deep dive in the code might be needed. TBC...