Go Down

Topic: Streamlining serial comms (with binary?) (Read 365 times) previous topic - next topic

ardvino

Hi,

I'm sending serial data from one Ardunio to another and need to reduce the bit-length of the string in order to communicate over a very slow underwater modem.

Currently I have three variables (A, B and C) with values being sent as numerical ASCII values (from 0-9) with alphabetical IDs. The strings are separated by [CR][LF] and look something like this, for example:

A3B6C2\r\n

These 8 characters represent 64 bits. The modem has a datarate of 100bps so I am only able to send fewer than 2 strings per second.

The receiving end parses the values and writes them to corresponding ints, which are then mapped and output to three corresponding analog outputs.

My question is; how might I streamline this communication to reduce the bit-length of the string. I wondered if I could send HEX or even binary instead of the rather cumbersome ASCII. If I were to send binary, for example (using Serial.write(val)), how would I read and distribute the three variables to ints at the receiving end? Can I use Serial.read()? I guess Serial.parse isn't going to work with anything other than ASCII.

Many thanks in advance! Below is my current receiving-end Arduino code.

Code: [Select]
int varA = 0; // int to hold received variable A
int varB = 0; // int to hold received variable B
int varC = 0; // int to hold received variable C

const int Apin = 3; // define pin 3 'Apin'
const int Bpin = 5; // define pin 5 'Bpin'
const int Cpin = 6; // define pin 6 'Cpin'

void setup() {

  Serial.begin(9600); // begin serial at 9600baud
  pinMode(Apin, OUTPUT); // set Apin as output
  pinMode(Bpin, OUTPUT); // set Bpin as output
  pinMode(Cpin, OUTPUT); // set Cpin as output
}

void loop() {
  if (Serial.available()) {
    if (Serial.read() == 'A') varA = Serial.parseInt(); // wait for ID character 'A' in string and then read following number into varA
    if (Serial.read() == 'B') varB = Serial.parseInt(); // wait for ID character 'B' in string and then read following number into varB
    if (Serial.read() == 'C') varC = Serial.parseInt(); // wait for ID character 'C' in string and then read following number into varC
  }
  analogWrite(Apin, (map(varA,0,9,0,255))); // map received value in varA to 0-255 and write pwm to output pin Apin
  analogWrite(Bpin, (map(varB,0,9,0,255))); // map received value in varB to 0-255 and write pwm to output pin Bpin
  analogWrite(Cpin, (map(varC,0,9,0,255))); // map received value in varC to 0-255 and write pwm to output pin Cpin
  }

sherzaad

#1
Nov 29, 2018, 04:45 pm Last Edit: Nov 29, 2018, 05:26 pm by sherzaad
you could use serial.write() to sent a "raw" byte

1 byte is 8 bits, then maybe the lower nibble would be your value (0-9) and the upper nibble the variable (A, B,C). you could potentially also ditch the \r and \n, but you need to be sure weary of any side effect then

so for example
serial.write(0xA3);
serial.write(0xB4);
serial.write(0xC6);

if you really wanted to be fancy, you could introduce a delay between transmission so that on the receiver side you could code something like this:
if previous received msg time - new recieved msg time> x then NEW command set
else 'new recieved msg' part of current command request

ardvino

Thank you sherzaad - I'd not thought of ditching the [CR][LF] - that will speed things up straight away.

Sending the values as pairs of nibbles is also a great idea. How would I read that string back into ints on the receiving arduino?

Many thanks

sherzaad

#3
Nov 29, 2018, 04:57 pm Last Edit: Nov 29, 2018, 05:01 pm by sherzaad
Thank you sherzaad - I'd not thought of ditching the [CR][LF] - that will speed things up straight away.

Sending the values as pairs of nibbles is also a great idea. How would I read that string back into ints on the receiving arduino?

Many thanks
break the byte! :)

so in pseudo code:

if serial.available, byte in = serial.read

uint8_t var1 = in & 0xF0; //gets upper nibble value only
uint8_t val = in & 0x0F;   //gets LOWER nibble value only

if var1 =0xA0, analogWrite(Apin, (map(val,0,9,0,255)));

else if var1 = 0xB0 ... you get the gist for B and C! ;)

end if
end if

Robin2

#4
Nov 29, 2018, 05:46 pm Last Edit: Nov 29, 2018, 05:47 pm by Robin2
Currently I have three variables (A, B and C) with values being sent as numerical ASCII values (from 0-9) with alphabetical IDs. The strings are separated by [CR][LF] and look something like this, for example:

A3B6C2\r\n
If each of the 3 values is always contained in a single byte then there is no need to send the A, B and C as the order of the bytes will identify them.

I would keep either the \r or the \n as a marker to identify the end of one message and (by implication) the start of the next.

If you start combining the nibbles of two bytes into one byte you may by accident come up with a byte value that is the same as (say) \n which will cause confusion. There are various stratagems for getting around that.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ardvino

if serial.available, byte in = serial.read

uint8_t var1 = in & 0xF0; //gets upper nibble value only
uint8_t val = in & 0x0F;   //gets LOWER nibble value only

if var1 =0xA0, analogWrite(Apin, (map(val,0,9,0,255)));

else if var1 = 0xB0 ... you get the gist for B and C! ;)

end if
end if

Gottit, thanks! I'll give this a go! Very much appreciated :-)

ardvino

#6
Nov 29, 2018, 11:23 pm Last Edit: Nov 29, 2018, 11:24 pm by ardvino
Thanks Robin2 :-)

If each of the 3 values is always contained in a single byte then there is no need to send the A, B and C as the order of the bytes will identify them.
Ok, yea this would also really cut down the bit-length of the string. So I could just send, for example:

362\r\n

which is 40 bits - not too bad.

So in my orignal code, I'm using the Serial.parseInt() function to split the values into A, B and C. Without the ID characters, how do I separate 362\r\n into A=3, B=6 and C=2? This might be a really noob question but how do I read just a single byte at a time? Would I just use Serial.ReadBytes(byte[],1)?

Splitting the bytes would get me down to 40 too, unless I also strip off the [CR][LF], in which case I'd be down to just 24!

cattledog

#7
Nov 30, 2018, 07:28 am Last Edit: Nov 30, 2018, 08:28 am by cattledog
I think you can do this by sending only two bytes. Consider them as 4 nibbles where the first nibble is a start marker, and the next three are your A,B,C values.  Since the values are limited 0-9, the start nibble can be a unique value greater than 9. I'd use 1111. You can use masks to separate the nibbles.
You will need to use the binary representations of 0-9

You will need these variables

Code: [Select]
boolean started = false;
byte receivedData[3] = {0,0,0};
byte inbyte;


Then the reading routine looks like this
Code: [Select]
if (Serial.available() && !started)
  {
    inbyte = Serial.read();
    if (inbyte & 0xF0 == 0xF0) //high byte is start marker B1111
    {
      receivedData[0] = inbyte & 0x0F; //low byte is data 0-9 B0000 thru B1001
      started = true;
    }
  }

  if (Serial.available() && started)
  {
    //next byte is data
    inbyte = Serial.read();
    receivedData[1] = inbyte & 0xF0;
    receivedData[2] = inbyte & 0x0F;
    started = false;
  }


So, if the A value is 3 and the B value is 6 and C value is 2 you will send two bytes
B11110011 and B01100010




Robin2

So, if the A value is 3 and the B value is 6 and C value is 2 you will send two bytes
B11110011 and B01100010

IMHO there should also be an end-marker byte with a value that can NEVER be in either of the data bytes. Without an end-marker there is a serious risk that the receiver gets confused about which byte is the first byte - for example if one byte gets missed.

362\r\n
which is 40 bits - not too bad.
Leave out the \n and save another 8

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

cattledog

#9
Nov 30, 2018, 05:17 pm Last Edit: Nov 30, 2018, 05:19 pm by cattledog
t
Quote
Without an end-marker there is a serious risk that the receiver gets confused about which byte is the first byte - for example if one byte gets missed.
There is less risk given that all transmissions are two bytes.

If the reception is in the not started state, and the byte with the unique start nibble is missed, no data will be accepted until the valid start byte is received. So missing a byte when in the not started state is benign.

If  the reception is in the started state and the second byte with the two data nibbles is missed, the code will skip the next start byte, but accept the next data byte received. Obviously this could be a problem.

I think that things could be made more robust, still working with nibbles, using a three byte transmission scheme with a unique end nibble say 1110. You also can check that there is one byte received between the bytes containing the start and stop nibbles.

You could even work a check sum into a three byte transmission where

Start byte is  1111DataA
Second byte is DataBDataC
Third byte is Checksum1110

A simple check sum is (DataA + DataB + DataC) %10.

If the start byte is missed, the reception does not start.

If the end byte is missed and the second date byte is read again on the next transmission, the check sum might be wrong and the total number of received bytes between start and stop will be greater than 1.

If the middle data byte is missed, then again the check sum might be wrong and the number of received bytes between start and stop will be 0 and not 1.


Robin2

If the reception is in the not started state, and the byte with the unique start nibble is missed,
I had missed the fact that there is a start-marker. With that there is obviously no need for an end-marker.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

ardvino

Thank you all very much. I have a much greater understanding now and I've done some reading on bitwise operators, following your suggestions.

I'm going to work on the comms this week and will report back.

ardvino

Ok guys, I've got much further on. Thank you again for all the advice.

I now just have one little hitch which hopefully someone can help with.

I am now using an XBOX controller to generate the 0-9 values. I have opted to go for a 2 byte message which contains a start nibble and three 0-9 values. So, my message format is like this:

0xF3 0x62

which translates to:

Start bit (F) (which is 1111 0000)
Value A = 3
Value B = 6
Value C = 2

I am managing to combine the bits ok, and getting the desired numbers out of the serial monitor. With no inputs on the controller, all values default to 4 (half way between 0 and 9) so I normally get:

F444
coming out of my serial monitor.

The section of code which generates this is as follows. I have not included all the USB host or XBOX code as I don't think it's relevant, but let me know if it would be helpful to post the whole lot. I have included a linefeed for the moment just to help me during debugging. startNibble = 0xF0

Code: [Select]
    Serial.print(startNibble | (map((Xbox.getAnalogHat(RightHatX)), -32000, 32000, 0 , 9)),HEX);
   
    bNibble = (map((Xbox.getAnalogHat(RightHatY)), -32000, 32000, 0 , 9)); // gives a value 0-9, with default from controller being 4, or 0000 0100
    bNibble = (bNibble << 4); // shifts mapped value from controller up 4 bits, so 4 or 0000 0100 becomes 0100 0000

    cNibble = (map(((0 - (Xbox.getButtonPress(L2))) + (Xbox.getButtonPress(R2))), -255, 255, 0, 9)); // gives a value 0-9, with default from controller being 4, or 0000 0100
   
    Serial.print(bNibble | cNibble, HEX); // combines and prints bNibble with cNibble to create a byte. So 0000 0100 | 0100 0000 becomes 0100 0100

    Serial.println();


This code works almost perfectly and the bitshifting seems to work. However, there's a problem with the second value (third nibble in the string, 'bNibble'). It's fine as long as its value doesn't equal 0. If the value is zero, it disappears from the string, so instead of seeing F404, I see F44.

Is this something wrong with my code? Something odd about the way arduino handles 0000 0000? or maybe just a quirk with the serial monitor? I'd really appreciate any thoughts on this.

Many thanks in advance! We're getting there!

sherzaad

This code works almost perfectly and the bitshifting seems to work. However, there's a problem with the second value (third nibble in the string, 'bNibble'). It's fine as long as its value doesn't equal 0. If the value is zero, it disappears from the string, so instead of seeing F404, I see F44.

Is this something wrong with my code? Something odd about the way arduino handles 0000 0000? or maybe just a quirk with the serial monitor? I'd really appreciate any thoughts on this.

Many thanks in advance! We're getting there!
if the HEX value of a byte is '04' for example, then serial.print will print it as '4'. i.e YOU would need to add the zero padding

cattledog

Quote
maybe just a quirk with the serial monitor?
The serial monitor does not print leading zeros.

Code: [Select]
void setup() {
  Serial.begin(115200);
  byte startNibble = 0xF;
  byte nibbleA = 0x4;
  byte nibbleB = 0x0;
  byte nibbleC = 0x4;
  Serial.println(startNibble << 4 | nibbleA, BIN);
  Serial.println(nibbleB << 4 | nibbleC, BIN);
  Serial.println((unsigned int) startNibble << 12 | nibbleA << 8 | nibbleB << 4 | nibbleC, BIN);
  Serial.println(startNibble << 4 | nibbleA, HEX);
  Serial.println(nibbleB << 4 | nibbleC, HEX);
  Serial.println((unsigned int) startNibble << 12 | nibbleA << 8 | nibbleB << 4 | nibbleC, HEX);
}

void loop() {}


Output:
Code: [Select]
11110100
100
1111010000000100
F4
4
F404

Go Up