UART radio project, advice appreciated

Hi,

I’m trying to create UART radio using nrf24l01+ and Pro mini. Idea seemed pretty straightforward, but obviously is not.

The purpose is to transfer RTCM3 messages from one Sparkfun zed-f9p (further zed) to another. It is said in docs that all you need is simple uart tx → rx. What happens there (according to my scope) is several messages are packed into one data packet lasting ~80-90ms and pushed to zed’s uart tx. Content of the messages on zed is known, but obviously zed does something while packing messages into the packet. I think so because what I see on Arduino IDE serial monitor when connect zed tx to Pro mini tx does not correspond to what I see on zed’s management sw output. In other words, packets on serial are not recognisable as the same on zed. I thought, whatever, I’ll just push through the radio whatever comes in and the other zed will deal with it. That’s how it’s supposed to work according to zed’s tutorial.

My idea was to give whatever comes from zed 1 tx to Pro mini’s software serial, read it into an array, then give the data to nrf, using RF24 and RF24Network libraries. On rx end, get the packet and write it to Pro mini 2 soft serial which would be connected to zed 2 uart rx.

In order to test the idea I took the simplest read soft serial example and modified it into the following. This is not the first version, there were many. This is what I have at the moment of writing the post, probably best I had so far:

#include <SoftwareSerial.h>

SoftwareSerial mySerial(2, 3); // RX, TX

int ndx = 0;
char Igotstring[680];
char rc;

void setup()
{
  Serial.begin(115200);
  Serial.println("Hello world");

  mySerial.begin(57600);
  mySerial.setTimeout(10);
}

void loop() {

if (mySerial.available()) {
  rc=mySerial.read();
  Igotstring[ndx]=rc;
  ndx++;
  
  } else if (ndx>0) { // I have not understood yet why without this extra I get a lot of empty "packets" between "real packets"
    Serial.print("Packet size: ");Serial.println(ndx+1);
    Serial.println("Packet:");
    for (int i=0;i<ndx;i++) Serial.print(Igotstring[i]);
    Serial.println(" EOP!");
    ndx=0;
    
  }

}

I get the packets printed on serial, but those are obviously shorter than expected. But longer than soft serial buffer (still 64?).

Now, it seem that I have not enough knowledge to use Arduino soft serial properly or I’m doing something totally impossible. Regarding knowledge, I really don’t. I’ve made a few projects, those are working nicely, but banging the serial for the first time. I have couple thoughts what could be wrong:

  • Using timeout for recognising the packet ends. I understand that is not the right way, but there is no start and no end markers, only timing. And, I don’t think I can send every byte over nrf separately, there should be some lag, however short which would break the packet. I tried something like that already, didn’t work.
  • Wrong datatypes and Arduino doing some magic with data as a result. I tried char and byte.
  • Packets are somehow “not continuous”… This would be the worst, this approach would be done. Meanwhile, my scope kind of says the opposite, I don’t see breaks in packets. However, the scope’s uart triggering does not recognise a lot of incoming data as data… this made me thinking…

I also tried to put while cycle int the code assuming it will read the whole packet during one loop(). Does not work at all.

Any advice would be appreciated.

Can You split this mammouth task into smaller, more handy parts, You would be better off. Else You will be drowning in replies attacking any part of the project.

Thanks for the hint. I tried to address only software serial read problem. Sorry, it was not obvious. Particularly, why the sketch could be "cutting" the packets short. The rest was for the folk to understand the context.

Tbh, if somebody has tried Arduino+nRF for tx/rx for long packets and it did not work out, I’d seriously listen about why.

For asynch to packet conversion, it is the transmitter that needs a time out. The receiver just unpacks and relays data to the end application immediately.

Software Serial is by far the worst of the options, and is error prone at Baud rates above 38400.

I suggest to try NeoSWserial or AltSoftSerial.

jtupulis:
The purpose is to transfer RTCM3 messages from one Sparkfun zed-f9p (further zed) to another.

[...]

My idea was to give whatever comes from zed 1 tx to Pro mini's software serial, read it into an array, then give the data to nrf, using RF24 and RF24Network libraries. On rx end, get the packet and write it to Pro mini 2 soft serial which would be connected to zed 2 uart rx.

There is a huge amount of detail that I am unaware of because I have no knowledge of RTCM3 or zed-f9p.

An obvious question is how many bytes are in a message and how many messages are produced per second.

If high throughput is needed then you need Hardware Serial and a high baud rate and you probably don't want to be using the RF24 Network as it adds complexity on top of the basic nRF24 library.

If you are planning to use nRF24s then I strongly suggest that you get that working on its own with one of the examples from my Tutorial and then add the other code that your project requires.

...R
Simple nRF24L01+ Tutorial
Serial Input Basics - simple reliable non-blocking ways to receive data. Can easily be adapted to work with SoftwareSerial

Robin2:
There is a huge amount of detail that I am unaware of because I have no knowledge of RTCM3 or zed-f9p.

Yeah, same here, I'm just learning that dev board and everything attached. But, for now I'm following zed tutorial and I'm stuck at the step which basically says: connect two zed's with uart radio at 57600 baud.

An obvious question is how many bytes are in a message and how many messages are produced per second.

From zed management SW I can see that the content of one packet sent over uart consists of 4-5 RTCM3 messages, ~100bytes each, ~400-500 bytes of information in total. As I said, those messages are not recognisable on the serial, I assume zed does some encoding. On the scope I can see that a packet takes ~80-90ms to transmit. At 57600, 8 bit data, no parity it should be ~460-520 bytes of data, which corresponds very well to data amount in RTCM3 messages. A packet is sent once per second.

If high throughput is needed then you need Hardware Serial and a high baud rate and you probably don't want to be using the RF24 Network as it adds complexity on top of the basic nRF24 library.

Yes, it will be hardware serial at the end. I'm using soft serial because I want to capture the data on computer and I have to have debug messages. Maybe I should switch to use soft serial for debugging...

RF24Network was my first idea because data packets are bigger than nrf24l0 buffer. I already recognised this library adds too much overhead and that I'll have to disassemble/assemble packets for nrf myself.

Thanks for your tutorials, I used those to learn and that was helpful in order to start!

jremington:
Software Serial is by far the worst of the options, and is error prone at Baud rates above 38400.

I suggest to try NeoSWserial or AltSoftSerial.

Yeah, I need 57600. I’ll look into the alternatives you mentioned. Thanks for the hint!

aarg:
For asynch to packet conversion, it is the transmitter that needs a time out. The receiver just unpacks and relays data to the end application immediately.

The sketch is a beginning of a transmitter. However, I think you mean the receiving from zed, the external source? If yes, then I can think of no other way than a timeout to finish a packet.

jtupulis:
Maybe I should switch to use soft serial for debugging...

I suggest you get a Mega which has extra HardwareSerial ports and do your initial development and testing with that.

...R

jtupulis:
The sketch is a beginning of a transmitter. However, I think you mean the receiving from zed, the external source? If yes, then I can think of no other way than a timeout to finish a packet.

No. Properly constructed packets do not require any RX time out, except for an RX error time out (e.g. to handle incomplete packets). That is because packets always have some kind of recognizable beginning and end, by design.

What I mean is, a time out is required for any transmission that does not fit in an integral number of packets. For example, If the packets contain 10 data bytes, and I start constructing and sending packets, I will begin filling the first packet with the first 10 bytes that I have to send, if they are immediately available. Then I send it, and start filling the next packet. This will straightforwardly convert a stream to packets. But if the stream is interrupted (which can happen due to the end of message in addition to communication delays) there will be unsent bytes in the last packet. If I send 25 characters, 20 of them will be sent immediately in 2 packets of 10 bytes. But the remaining 5 bytes need to be sent. This is what the TX time out is for. If no more data is available to continue filling the buffer, it will time out and send the 5 bytes. Alternatively, if there is some processing delay and any continuous stream is paused, it will also time out and send whatever it has in the buffer. That way, the receiver will have the most current data possible, even though it is unknown when the next data will arrive.

In this way, and possibly in no other way, it is possible to almost transparently convert a stream to a series of packets.

Note that in many situations, there is no encoding in the input stream that can inform the packet constructor that an end of message is reached. Thus it has to do timing.

Robin2:
I suggest you get a Mega which has extra HardwareSerial ports and do your initial development and testing with that.

…R

Yes, you are right. I just ordered one.

Meanwhile, I rewrote the simple sketch above to use hw serial for incoming data and sw serial for debug output. It is a little messy with wires reconnecting for every code update, but it seem to be working :slight_smile:

#include <SoftwareSerial.h>

SoftwareSerial mySerial(2, 3); // RX, TX

int ndx = 0;
char Igotstring[680];
char rc;

void setup()
{
  Serial.begin(57600);
  Serial.setTimeout(100);

  mySerial.begin(115200);
  mySerial.println("Hello world");
}

void loop() {

if (Serial.available()) {
  rc=Serial.read();
  Igotstring[ndx]=rc;
  ndx++;
  
  } else if (ndx>0) { // I have not understood yet why without this extra I get a lot of empty "packets" between "real packets"
    mySerial.print("Packet size: ");mySerial.println(ndx+1);
    mySerial.println("Packet:");
    for (int i=0;i<ndx;i++) mySerial.print(Igotstring[i]);
    mySerial.println(" EOP!");
    ndx=0;
    
  }

}

Output is broken into pieces, but pieces make up for ~400 bytes, which is about right. Also, those broken pieces are incoming once per second, which is also right. Now I have to figure out what breaks the packet into pieces. Most probably, I have to understand what and when triggers !Serial.available()

Debug output of one particular packet:

Packet size: 2
Packet:
⸮ EOP!
Packet size: 24
Packet:
�PC �PH⸮⸮����⸮!���� �⸮� EOP!
Packet size: 40
Packet:
N⸮⸮"⸮⸮y죇t⸮42w⸮⸮⸮?⸮⸮$⸮⸮⸮⸮⸮⸮⸮⸮D EOP!
Packet size: 51
Packet:
⸮�������������������
%w⸮�AC⸮��t⸮���⸮���� ⸮��v⸮⸮ EOP!
Packet size: 58
Packet:
⸮⸮⸮⸮⸮..TF⸮u⸮k⸮⸮⸮⸮⸮r⸮⸮+=⸮⸮O⸮⸮⸮⸮⸮Q������⸮Wp⸮�{D`�PH⸮⸮��P EOP!
Packet size: 63
Packet:
!@��� ��wڠ⸮⸮⸮⸮⸮⸮⸮~⸮⸮2;⸮.|AZ⸮⸮⸮⸮E⸮⸮⸮w⸮⸮'⸮<c⸮⸮⸮⸮⸮$�⸮L⸮a EOP!
Packet size: 66
Packet:
^⸮⸮3⸮叆|⸮⸮}8>⸮⸮⸮⸮⸮⸮⸮�Xې9⸮\⸮������������������X&⸮�^F⸮�PG⸮�� EOP!
Packet size: 68
Packet:
��⸮�⸮��� ��-Po⸮⸮⸮a`⸮OH⸮⸮C8⸮ur⸮j⸮⸮⸮[⸮=h⸮7⸮�⸮⸮6⸮����������� EO⸮!
Packet size: 22
Packet:
������������������{>3 EOP!

aarg:
No. Properly constructed packets do not require any RX time out, except for an RX error time out (e.g. to handle incomplete packets). That is because packets always have some kind of recognizable beginning and end, by design.

Thanks for the reply!

I understand the theory. In my case there are obstacles. First is, packets are in some way encoded. I can’t recognise RTCM3 messages in them. Second, even if I could, messages do not have end marker and are of variable length. Most probably, zed is “packaging” messages in some way, most probably in a logical way where a packet has start and end markers sp the other zed can “read” a packet. But I have not figured out a way to get to know those markers. What I have tried this: I redirect incoming from zed to a file and tried to search for random pieces of messages there. There are none. So, currently I’m stuck with “I have to transfer whatever comes in”…

I’m adding an example how two consequent packets’ contents are looking from zed software. And an example of how it looks coming out via serial.

Contents of a packet, consists of 4 messages:

11:34:28  0000  D3 00 5E 43 20 00 33 23 3F 66 00 00 04 05 20 02  Ó.^C .3#?f.... .
          0010  00 00 00 00 20 00 80 00 77 E9 08 EA 2A 08 82 DE  .... ...wé.ê*..Þ
          0020  32 5A FE 04 A6 01 3B AF 3A EE FF C7 7D F1 23 6E  2Zþ.¦.;¯:îÿÇ}ñ#n
          0030  3F 1F 57 96 9E 9E F8 E2 7B AC 28 F6 18 C3 FB E8  ?.W...øâ{¬(ö.Ãûè
          0040  9F F0 4F 82 A3 CE 08 47 37 D7 FE BF 54 16 3B BB  .ðO.£Î.G7×þ¿T.;»
          0050  BB BB AC 01 0C 57 CF 3B 73 C4 E8 00 00 00 00 00  »»¬..WÏ;sÄè.....
          0060  00 2C 2D C8                                      .,-È.
          
11:34:28  0000  D3 00 48 43 C0 00 4C 82 74 26 00 00 0E 01 03 00  Ó.HCÀ.L.t&......
          0010  00 00 00 00 20 00 00 00 7E 8E 80 90 82 98 9B F1  .... ...~......ñ
          0020  8E 7C 71 7F 84 59 C2 6C CD BA 88 EB 24 B0 73 CF  .|q..YÂlͺ.ë$°sÏ
          0030  9C 08 23 0C 21 B6 AC 02 65 66 11 6D 00 72 B5 5F  ..#.!¶¬.ef.m.rµ_
          0040  35 0A 3A F6 75 01 13 B5 39 4B C0 5E 23 EB        5.:öu..µ9KÀ^#ë.
          
11:34:28  0000  D3 00 7B 44 60 00 33 23 3F 66 00 00 09 10 14 10  Ó.{D`.3#?f......
          0010  08 00 00 00 20 01 00 00 6B AE BC B8 AA 9E AA 9C  .... ...k®¼¸ª.ª.
          0020  B8 85 3F 7F 52 F7 9D E6 6F 76 0A 9B FA BF 2C CC  ¸.?.R÷.æov..ú¿,Ì
          0030  A4 89 AA 11 0E 47 E0 48 71 F9 13 DE DD F5 43 F7  ¤.ª..GàHqù.ÞÝõC÷
          0040  FD D9 FC B0 B8 64 C5 A0 98 ED 82 19 2E 11 FA 30  ýÙü°¸dÅ .í....ú0
          0050  26 10 A1 F9 21 07 BD F1 BB 37 DD B9 DC 00 49 5B  &.¡ù!.½ñ»7ݹÜ.I[
          0060  70 4D 5C AF CF 40 00 00 00 00 00 00 00 00 00 00  pM\¯Ï@..........
          0070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 66 9C  ..............f.
          0080  3C                                               <.
          
11:34:28  0000  D3 00 50 46 40 00 33 22 64 A4 00 00 00 30 02 00  Ó.PF@.3"d¤...0..
          0010  08 00 00 00 20 02 00 00 3D 3E AC 27 A9 8A 20 C2  .... ...=>¬'©. Â
          0020  1E 00 75 89 16 5E 20 50 6E DE 1A AF AC 67 01 53  ..u..^ PnÞ.¯¬g.S
          0030  A7 03 D2 F8 1C 3A 2F 17 2F 6B BB B4 0F 43 3E 0E  §.Òø.:/./k»´.C>.
          0040  80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
          0050  00 00 00 D1 51 DB                                ...ÑQÛ.

Contents of the next packet, one second later, 5 messages:

11:34:29  0000  D3 00 5E 43 20 00 33 23 4F 06 00 00 04 05 20 02  Ó.^C .3#O..... .
          0010  00 00 00 00 20 00 80 00 77 E9 08 EA 2A 08 82 E6  .... ...wé.ê*..æ
          0020  2E 59 FE 04 A6 20 6B ED 20 B8 32 B0 E3 C4 28 08  .Yþ.¦ kí ¸2°ãÄ(.
          0030  48 4F BA AF 63 EF 08 8A FB EA C6 01 AB C7 C7 93  HOº¯cï..ûêÆ.«ÇÇ.
          0040  5F 1E FB 02 EC D5 09 6B 37 F0 9F 4F B6 8E BB BB  _.û.ìÕ.k7ð.O¶.»»
          0050  BB BB AC 01 0C 37 CF 3B 93 C0 F8 00 00 00 00 00  »»¬..7Ï;.Àø.....
          0060  00 89 C0 3B                                      ..À;.
          
11:34:29  0000  D3 00 48 43 C0 00 4C 82 83 C6 00 00 0E 01 03 00  Ó.HCÀ.L..Æ......
          0010  00 00 00 00 20 00 00 00 7E 8E 80 90 82 98 9B F3  .... ...~......ó
          0020  0E 9C 61 7D 84 D9 9A 0E 35 C3 EE B5 11 AE 79 74  ..a}.Ù..5Ãîµ.®yt
          0030  09 0F 9F 37 5E B6 FB 7B D7 A2 07 ED 00 78 63 C0  ...7^¶û{×¢.í.xcÀ
          0040  0F 9E 3B 76 75 01 13 D5 39 4C 80 9A 1D B1        ..;vu..Õ9L...±.
          
11:34:29  0000  D3 00 7B 44 60 00 33 23 4F 06 00 00 09 10 14 10  Ó.{D`.3#O.......
          0010  08 00 00 00 20 01 00 00 6B AE BC B8 AA 9E AA 9C  .... ...k®¼¸ª.ª.
          0020  B8 84 3F BF 5A F5 9C E6 6F 76 67 DC B4 5E B1 CA  ¸.?¿Zõ.æovgÜ´^±Ê
          0030  1F 1C 5B 36 7A 7A 50 00 67 6E 1E C8 BE 52 51 F9  ..[6zzP.gn.ȾRQù
          0040  72 1D FA 97 B0 50 71 21 C4 36 06 C6 36 1E 95 20  r.ú.°Pq!Ä6.Æ6.. 
          0050  02 62 7F 6E 3B 7D 92 4F BB 37 DD B9 DC 00 49 5B  .b.n;}.O»7ݹÜ.I[
          0060  90 4D 5C AF 4F 40 00 00 00 00 00 00 00 00 00 00  .M\¯O@..........
          0070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 3D 6F  ..............=o
          0080  98                                               ..
          
11:34:29  0000  D3 00 50 46 40 00 33 22 74 44 00 00 00 30 02 00  Ó.PF@.3"tD...0..
          0010  08 00 00 00 20 02 00 00 3D 3E AC 27 A9 0A 20 CA  .... ...=>¬'©. Ê
          0020  1D FF 6F 0A D8 CD A5 3B C5 46 02 5F 78 60 BD 7A  .ÿo.ØÍ¥;ÅF._x`½z
          0030  D3 F4 6F DB F1 DF 4F 0B 7B 6B BB B4 0F C3 3E 0D  ÓôoÛñßO.{k»´.Ã>.
          0040  80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
          0050  00 00 00 9B E2 39                                ....â9.
          
11:34:29  0000  D3 00 04 4C E0 00 80 ED ED D6                    Ó..Là..ííÖ.

A packet how it looks on serial:

Ó�§C �UI!B��Ap
���� �€�}}ZJjRR‚*jR*-©’¸äÄuäe~y-ð<@˜v‘¢ÓPû±6³†¹
RÏ÷>²¯]ž#üŽ46�uùƒ?
“ÝIïl¢€ß|(é_õ¸}W—ýp'óooÿÿÿÿÿÿÿ�…CUuÉusÕeÖ�������������������������������'éâÓ�{CÀ�…uv��aA€���� €��_þžŒ€’ŒŒ)­=l£œi‘Z¾ý=ñU‚`,ÄTÈúQ଼8zîtç
`×ïµ¹ó�N;,Ô-ЫÉCìŸa§Ëê
)¾¾ØúŒ‡ÿÿÿÿÿÿ�RÝe”_VPð������A3Ó�{D`�UI!B���Á@��� ��þ¶˜” °³u
+hãáe¾¦S4{‰õ˜kÌè¿={ @A’"šåWøÁ™0†-Gûâçêba®„a:ñ뾯ª@å�Œ[ÀE¦_ÿÿÿÿÿÿà�Sm5Ó]wWuuO0Vñ¤Ó�/F@�UHF€��������� ���u$ԏÐ_m3öP,wèÀR/ábmÿø¦ºÀÒ˜»Ó�>Ð�¼ÿUZεÞèZ	€4ð¤öXN

Just to get a feel of how the signal looks on the scope in several time scales, drilling down: album

An update.

I just got Mega board. Now I can work with hardware serial for development. Cool :slight_smile:

But the problem is the same. Maning, simple read from one hw serial - write to another hw serial still breaks the data into chunks.

I got a step further. I can see, for example, how sketch does breaking. Just look at the pic from scope (last pic in the album). Clearly, Arduino perceives that serial is NOT available while receiving 0x00... wtf... Maybe this is again something obvious?

Btw, I also recognised that probably zed board is not encoding the data, because I can see that data stream received on Arduino always starts with 0xd300, same as on zed. So, I've got a start marker, at least. Will rewrite sketch.

It looks like you are breaking packets based on Serial.available. The Arduino is fast enough that it can process a single character before another arrives, so available isn't telling you anything important.

It would help i you could find some documentation on the actual packet content.

wildbill:
It looks like you are breaking packets based on Serial.available. The Arduino is fast enough that it can process a single character before another arrives, so available isn't telling you anything important.

Thanks for the reply!
Is there another way to know when to start read? All examples I've seen propose to use available().

It would help i you could find some documentation on the actual packet content.

That's a problem for me at this stage. RTCM message standards are proprietary and download of a single document costs several hundred USD, there are several :frowning: What I have figured out is the approx message length (it's not constant) and that each message begins with 0xd300, that's all I have at the moment.

jtupulis:
But the problem is the same. Maning, simple read from one hw serial - write to another hw serial still breaks the data into chunks.

I haven't used NRF24L01 in a while, but it packetizes the serial data stream into chunks in ways that may not be obvious. Perhaps what you're seeing is just that characteristic of the radio.

My understanding is that it will buffer the serial data stream on the transmit side and send it either when there are enough bytes to fill a packet or when a particular time interval since the last transmission has passed and it sends a less-than-full packet.

Using Serial.available is fine. It's just that you can't necessarily assume that if nothing is available you have reached the end of a packet.

MrMark:
I haven't used NRF24L01 in a while, but it packetizes the serial data stream into chunks in ways that may not be obvious. Perhaps what you're seeing is just that characteristic of the radio.

Hi,
Yeah, I know I'll have to adapt to nrf24 buffer. But I haven't gotten into nrf24 part yet, I'm fighting with reading zed board data from Arduino serial correctly.