Developing own serial communication protocol for learning purposes

[Kerby]

Hey guys,

I could need some advice with my project mentioned in the subject.
I know, I could just use an already known serial communication protocol like I2C, but this is not my goal.
I would like to build up a communication between different microcontrollers without any already provided function
and understand which difficulties one could experience when trying to develop something like that.

Some ideas for the communication protocol are based on I2C and before posting code,
I would like to explain the basic idea. There are 2 arduinos (1x transceiver and 1x receiver) connected to each other through 4 cables. The receiver is an Arduino Mega 2560 from Elegoo and the transceiver is an Arduino Uno with Atmega328P-PU.

Two cables represent SCL and SDA. One is connecting ground of both of them and another one is just for debugging purposes and acts like a status pin. The status pin tells the transceiver, when it has to send a new message.

These cables are directly connected to arduinos.
I would like to change this in the future by using pullup resistors with an external voltage source but not yet.

So, let's have a look at how my messages are constructed at the moment (attachment "signal_message_BITS.PNG"):

As you can see, a new message always starts by a falling edge. Before that SCL and SDA were HIGH.
Afterwards SCL is always changing whenever a new bit should be read. A message has a size of 8 bits.

Now let's have a look on the whole code for the receiver part.
The transceiver is just sending signals in the above described pattern. Therefore I have not considered to post this code, too.
But, if this would be interesting for you, just let me know and I will post it afterwards immediately.
I don't want to exaggerate the topic already at the beginning.

The receiver code:

bool recvSDA;
bool recvSCL;

unsigned long numMsg = 0;
unsigned long errorCounter = 0;

bool recvMsg = false;
byte counter = 0;
bool lastSCL;
bool isMsgReceived = false;

byte recvBits;
byte lastBits;

void setup() {
  Serial.begin(2000000);

  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);

  // Set status pin to zero at startup:
  DDRG  |=  B00100000;
  PORTG &= ~B00100000;
}

void loop() {
  
  if (recvMsg == false) {
    // If ready for a new message, set status pin to one:
    PORTG |= B00100000;

    // Since we are waiting for a new message, start a while loop for signal detection:
    while (1) {
      
      // Read out SCL and SDA with register manipulation:
      recvSDA = PINE & 1 << 3;
      recvSCL = PINH & 1 << 3;

      // As soon as a new message is recognized and incoming, prepare all required variables:
      if (recvSDA == HIGH && recvSCL == LOW && recvMsg == false) {
        counter = 0;
        ++numMsg;
        
        recvBits = 0;
        recvMsg = true;

        // The status pin can be set to low:
        PORTG &= ~B00100000;
        continue;
      }

      // This section is detecting all HIGH bits and saving them:
      if (recvMsg == true  && counter <= 7 && (recvSCL - lastSCL) != 0) {
        lastSCL = recvSCL;
        
        if (recvSDA == HIGH) {
          recvBits = recvBits | 1 << (7 - counter);
        }
        
        ++counter;
      }

      // As soon as 8 bits were received, start the end procedure for the message:
      if (recvMsg == true && counter > 7) {
        if (recvSCL == HIGH && recvSDA == HIGH) {
          if (isMsgReceived == false) {
            lastBits = recvBits;
            isMsgReceived = true;
          }

          // For debugging purposes we are sending the exact same message all the time.
          // Therefore changes between several messages should be zero.
          // If not, we are going to detect this by the following pattern:
          errorCounter += (lastBits != recvBits);
          recvMsg = false;
          lastBits = recvBits;

          // After a predefined number of messages, print out the status.
          // If you are doing it every cycle, the datarate will decrease a lot!
          if (numMsg % 10000 == 0) {
            Serial.print(numMsg); Serial.print(", "); Serial.print(errorCounter); Serial.print(", "); Serial.println(recvBits, BIN);
          }
          
          break;
        }
      }
    }
  }
}

I hope, it is readable for you. Just let me know, if you have advices for better readability.

As you can see, at the beginning of every loop(..) I'm setting the status pin to HIGH, so the transceiver should start sending a new message. Afterwards I'm starting an infinite while loop which is looping all the time to receive any transmitted signal.
I used the while loop instead of the loop(..) function of arduino to make sure, I'm not losing to much time between every new loop.
With the loop(..) function I have experienced that there is sometimes a big delay between two loops which are not always constant and disturbed the receiving procedure and led to loosing some bits. Therefore my decision to execute the signal detection with a while loop.

Though I have a feeling that using a while loop and trying to detect an incoming message with this pattern is probably not the best approach.
I have read about interrupts and their different detection possibilities, but I don't know if this is the fastest way to detect messages.
Is their any way to improve the speed of detecting new incoming messages? Are interrupts the fastest way of detection or is a while loop constructed like described above also a possible and fast way detecting new messages?

Let me know, if you have any advice for me!
I'm especially interested on hints which are explaining me some new aspects I have not considered till now.

Thank you for your time! I know, all of you are doing this in your freetime, therefore: every help is much appreciated.

Best regards,
Kerby

PS: I also uploaded the output in serial monitor (see "serialMonitor_output.PNG").
At the moment the communication works well, but I'm working on to improve the data rate of the system.

[DrDiettrich]

Do you really mean "transceiver" or simply "transmitter"?

Register manipulation may be good for machine speed but near impossible to read for humans.

How is your protocol different from a synchronous serial protocol with a NRZ clock?

Have you tried to transmit all possible 8 bit values?

How is your protocol better (faster...) than existing protocols?

[Klaus_K]

Kerby:
Let me know, if you have any advice for me!
I'm especially interested on hints which are explaining me some new aspects I have not considered till now.

If you are interested in creating new ways of serial communication I think a FPGA would be a much better tool to learn. Processors are far to valuable to waste on port pin bit manipulation. That is the reason why microcontroller have some many communication peripherals today.

With an FPGA you could learn how these circuits are made inside the chip. You could start with something simple and create a UART or SPI and then work your way up to more complicated protocols like CAN, RF or your own. This would also teach you about hardware languages like VHDL or Verilog.

[sterretje]

Why not make use of the available hardware in the processor? You don't have to use libraries and can manipulate registers directly to achieve what you want to.

That's the way I would go unless I have to implement a bit-banged I2C / Serial / SPI.

[Kerby]

DrDiettrich:
Do you really mean "transceiver" or simply "transmitter"?

Uuuh, you are right. Shame on me, I misunderstood the meaning of transceiver. It is a transmitter!

DrDiettrich:
How is your protocol different from a synchronous serial protocol with a NRZ clock?

To be honest: I'm not really sure what you meant with this question, since I don't know what a NRZ clock is.
But I have seen that there is an article on wikipedia about "Non-return-to-zero", which I will read through.

DrDiettrich:
Have you tried to transmit all possible 8 bit values?

Yeah, I changed the transmitter code in such a way, that it would send all values from 0 to 255 cyclical:

The picture visualises the received bytes on the receiving side.

I have not experienced any misbehaviour, but we also have to take into account, that Serial.println will be executed every time a new message arrives, which decreases the data rate a lot. Who knows how a higher data rate would influence the stability of the system.

DrDiettrich:
How is your protocol better (faster...) than existing protocols?

Absolutely not better or faster. As I already said, this is just a project which I would like use to learn something new and as we can see, you are all clearly giving me some new ideas or aspects I have not considered at all.
Before this project I have never heard about register manipulation, timers and interrupts which was quite nice to know.
That has led me to read the documentation about the used microcontrollers, but I'm still not done with and more importantly have not understood everything described in there.

Klaus_K:
If you are interested in creating new ways of serial communication I think a FPGA would be a much better tool to learn. Processors are far to valuable to waste on port pin bit manipulation. That is the reason why microcontroller have some many communication peripherals today.

With an FPGA you could learn how these circuits are made inside the chip. You could start with something simple and create a UART or SPI and then work your way up to more complicated protocols like CAN, RF or your own. This would also teach you about hardware languages like VHDL or Verilog.

Nice! Very valueable input from you! Will consider this and read through some topics articles regarding FPGA.
Correct me please, but from my basic understanding an FPGA is like a microcontroller, whose circuit can be programmed and changed in nearly any possible way, right? Besides this, does that mean, that e.g. our arduino microcontrollers like Atmega 2560 has already integrated a separate circuit only for communication?

sterretje:
Why not make use of the available hardware in the processor? You don't have to use libraries and can manipulate registers directly to achieve what you want to.

That's the way I would go unless I have to implement a bit-banged I2C / Serial / SPI.

Which available hardware exactly do you mean?
What are the required registers?
I guess, there is a connection between the second post and your post, since both of you are describing a specific hardware on the arduinos. Probably something integrated in the atmegas. I guess I could find more about this in the documentation of the atmegas.

"Bit-Banging" - good to know how my selfmade communication is usually described
There are also articles about the advantages and disadvantages of such an approach.

Thank you guys already for your time!
You see, that's the reason I'm asking here because without any input from experienced guys like you,
I would always do the exact same thing and now I can investigate more aspects I have not known before.

[Robin2]

Kerby:
As you can see, a new message always starts by a falling edge. Before that SCL and SDA were HIGH.
Afterwards SCL is always changing whenever a new bit should be read. A message has a size of 8 bits.

I don't see the need for a new message to start with a falling edge if you have a clock signal. In asynchronous Serial communication there is no clock signal and the falling edge is used to tell the receiver when to start its clock so that it is in sync with the transmitter.

With a clock signal I would expect the receiver to read the data line every time the clock transitions from (say) HIGH to LOW. Or, looking at it from the other end, the Tx would set the data line and then toggle the clock line from HIGH to LOW

...R

[DrDiettrich]

Don't worry about speed in the proof-of-concept phase. Finally the achievable transmission speed is proportional to the clock frequency. Highest possible speed requires short lines or special line drivers (differential, fiber optic...), independently from the bit level protocol.

I'd like to read more about the 4th wire (status pin). Such a signal is required in half-duplex transmission or in a network.

[Klaus_K]

Kerby:
Correct me please, but from my basic understanding an FPGA is like a microcontroller, whose circuit can be programmed and changed in nearly any possible way, right?

Yes, FPGA have an array of logic gates that can be used to create complex digital circuits. The more gates a FPGA has, the more complicated things you can create.

Kerby:
Besides this, does that mean, that e.g. our arduino microcontrollers like Atmega 2560 has already integrated a separate circuit only for communication?
..
Which available hardware exactly do you mean?
What are the required registers?
..
I guess, there is a connection between the second post and your post, since both of you are describing a specific hardware on the arduinos. Probably something integrated in the atmegas. I guess I could find more about this in the documentation of the atmegas.

The hardware circuits we are talking about are called peripherals. They are circuits inside the microcontroller that aid or completely handle the different communication protocols.
Some like USARTs, SPI and I2C are fairly simple and physically small inside the chip and you often will find many of them.
Others like CAN, USB are much more complicated and use more chip size, so they are less common.
For the latest radio protocols like BLE, WiFi the hardware is really complicated and some even have little hidden processors inside.
And these are just the most common communication peripherals. There are many chips that have specialized peripherals for proprietary protocols.

All of the these peripherals are configured and controlled by registers. This is usually hidden by software libraries. You can look at the source code and compare with the datasheets.

[DrDiettrich]

See the OSI model for layers of a communication protocol. The physical layer can be implemented in hardware, the remaining layers AFAIK are implemented in software (firmware).

[Kerby]

Robin2:
I don't see the need for a new message to start with a falling edge if you have a clock signal. In asynchronous Serial communication there is no clock signal and the falling edge is used to tell the receiver when to start its clock so that it is in sync with the transmitter.

With a clock signal I would expect the receiver to read the data line every time the clock transitions from (say) HIGH to LOW. Or, looking at it from the other end, the Tx would set the data line and then toggle the clock line from HIGH to LOW

...R

Good point.
I have done it that way, to make sure that the receiver knows exactly when a message is done.
For example, if I would reset the arduino and the receiver is not ready, it could start detecting bits in the middle of a message and all subsequent messages would be read wrongly, since it always starts and ends in the middle of a message. With my designed protocol I wanted to make sure, that always a whole message is received and not mixed with a new one.

It developed by the time to this manner, but start reading your post I think I would have to overthink my protocol. With the so called status or request pin I have mentioned in the past posts, this should likely not happen.

DrDiettrich:
Don't worry about speed in the proof-of-concept phase. Finally the achievable transmission speed is proportional to the clock frequency. Highest possible speed requires short lines or special line drivers (differential, fiber optic...), independently from the bit level protocol.

I'd like to read more about the 4th wire (status pin). Such a signal is required in half-duplex transmission or in a network.

Yeah, you are right, but my goal was in the beginning to understand what are the bottlenecks for data rate and when and why is this happening. I thought, maybe I can go as low-level as possible (from my understanding - I don't have any experiences with assembler) with an arduino to establish a data rate equal to i2c in standard mode (100 KBits/s).

At the moment my speed is around 30 KBits/s (and I'm not totally sure, if this is an approx. correct value)
and now I wanted to increase that before I start with the debugging and extra bits for address, parity, etc...
That was also the reason regarding the interrupts and how can one detect an incoming message as fast as possible.

Yeah, (16 MHz / required cycles for a message) is the limit, but I want to use that frequency as much as I can. If physical effects are limiting the speed, that is alright for me. The question then is, will I ever understand that the misbehaviour is caused by noise, overshooting or electrical intereferences without an oscilloscope or logic analyzer...

Regarding your status/request pin question, I uploaded the transmitter code:

const byte UnoSDA = 5;
const byte UnoSCL = 6;
const byte RequestPIN = 4;

bool laststate = LOW;
bool request;

int message = 256;

void setup() {
  Serial.begin(2000000);

  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);

  // Set SCL and SDA as output pins:
  DDRD  |= B01100000;

  // Set SCL and SDA to HIGH:
  PORTD |= B01100000;

  pinMode(RequestPIN, INPUT);
}

void loop() {

  while (1) {
    // Check, if receiver is waiting for a new message:
    request = PIND & 1 << RequestPIN;

    // If receiver is requesting a new message, start sending the message:
    if (request == HIGH && laststate == LOW) {
      SendSignalBinary(message);
      laststate = HIGH;
    }
    // This should make sure, that no new message will be sent
    // till the request pin was set to LOW again:
    else if (request == LOW && laststate == HIGH) {
      laststate = LOW;
    }
  }
}

void SendSignalBinary(byte message) {
  byte counter  = 0;
  byte index    = 0;
  byte duration = 0;
  bool flag     = 0;
  bool lastSCL  = LOW;
  byte stepSCL  = 3;
  byte stepSDA  = 2;
  int  BITS[8]  = {2, 5, 8, 11, 14, 17, 20, 23};

  while (counter <= 26) {
    ++counter;

    if (counter > BITS[index]) {
      ++index;
    }

    if (counter == 1) {
      digitalWrite(UnoSCL, LOW);
      continue;
    }
    else if (counter == 26) {
      digitalWrite(UnoSCL, HIGH);
      continue;
    }

    if (counter == stepSCL) {
      if (lastSCL == LOW) {
        digitalWrite(UnoSCL, HIGH);
        lastSCL = HIGH;
      }
      else if (lastSCL == HIGH) {
        digitalWrite(UnoSCL, LOW);
        lastSCL = LOW;
      }

      stepSCL += 3;
    }

    if (flag == 0) {
      if (counter == BITS[index] && (message & 1 << (7 - index)) > 0) {
        digitalWrite(UnoSDA, HIGH);
        flag  = 1;
        duration = BITS[index] + 2;
      }
      else if (counter == BITS[index] && (~message & 1 << (7 - index)) > 0) {
        digitalWrite(UnoSDA, LOW);
        flag  = 1;
        duration = BITS[index] + 2;
      }
    }
    else if (flag == 1) {
      if (counter >= duration && counter < 23) {
        if ((message & 1 << (7 - index)) > 0) {
          flag  = 0;
        }
        else {
          digitalWrite(UnoSDA, LOW);
          flag = 0;
        }
      }
    }
  }

  // Will be executed after the end of the while loop (after the 26th loop)
  // and sets the SDA to HIGH at the end of the message:
  PORTD |= B01100000;
}

The principle of the uploaded code can be easier described by the following picture:

One message is divided into 26 loops described by a while loop.
SCL is always changing one loop after the SDA was set. Every SDA bit has a duration of 3 loops.

Klaus_K:
The hardware circuits we are talking about are called peripherals. They are circuits inside the microcontroller that aid or completely handle the different communication protocols.
Some like USARTs, SPI and I2C are fairly simple and physically small inside the chip and you often will find many of them.
Others like CAN, USB are much more complicated and use more chip size, so they are less common.
For the latest radio protocols like BLE, WiFi the hardware is really complicated and some even have little hidden processors inside.
And these are just the most common communication peripherals. There are many chips that have specialized peripherals for proprietary protocols.

All of the these peripherals are configured and controlled by registers. This is usually hidden by software libraries. You can look at the source code and compare with the datasheets.

Thank you again! Nice to know that. I had no clue how this is usually implemented on µCs.
Good advice, to look for the libraries source code and compare with the datasheets!

DrDiettrich:
See the OSI model for layers of a communication protocol. The physical layer can be implemented in hardware, the remaining layers AFAIK are implemented in software (firmware).

Ahh I remember, I was confronted with this model in my working life. Always makes me happy if I see that something I'm doing in my freetime has even a little connection to my daily work.
As a side note: I'm working as a technical support engineer for a virtual test drive simulator software manufacturer in germany with a focus on ADAS and AD and started there 8 months ago after my study.