Sending data over serial in as few bytes as possible with fixed length.

Projects Programming
1

Im trying to get data from an arduino to a receptor in as few bytes as possible.
Im working with 1 int, 2 longs and 3 unsigned longs, with new line and return carriage chars at the end of each transmission. Which should total to 5*4 + 2= 22bytes for the data and 12bytes of returns and new lines.

However, just using Serial.println() ends up sending anywhere from 64 to 74 bytes of data in my tests, since it seems to be sending a byte for each figure on its own.

Is there a way to print through serial just 2 bytes for each int and 4 bytes for each long?
Or to combine all the variables in one big 22byte package and let the reception end deal with separating them again since they will be always printed out in the same order?

Abridged version of my code so far:

long latitude, longitude;
unsigned long age, resultado, date, time;
int session_id = 5;
void setup() {
 /*
 Some stuff happens here
 */

}

void loop() {
  /* Even more stuff */
  Serial.println(session_id);
  Serial.println(resultado);
  Serial.println(latitude);
  Serial.println(longitude);
  Serial.println(date);
  Serial.println(time);
}

EDIT: Nevermind I just realized that what I was looking for was Serial.write(), my math seems to be off somewhere tho because Im getting 39 bytes instead of 22.

2

Do you want to transmit binary data ? or as text (readable ASCII).

With binary data there could be a "\r\n" inside the data.
Different variables in a fixed format: that is a 'struct'.
It is possible to add a in front and a at the end.

For text, you could seperate the values with commas.
To make it a fixed length, you could add zeros in front of the numbers or convert everything to hexadecimal values.
The Serial.println() makes text from a binary variable. That is why more bytes are created.

Which Arduino boards do you use ?

3

I want to transmit binary data. I thought that using println would be easier to do while I worked but I was wrong on that.

For boards, Im using an arduino one as the sender of the data, and a raspberry pi with some python code as the receiver.

For now I changed the code in the arduino to

Serial.write(session_id);
  Serial.write(resultado);
  Serial.write(latitude);
  Serial.write(longitude);
  Serial.write(date);
  Serial.write(time);

And I am trying to make it so the raspberry decodes back again to integers, that is the part giving me trouble right now.

4

If you want to send binary, pack your data in a struct and send it in binary. You can use This Library or roll your own.

5 
long latitude, longitude;
unsigned long age, resultado, date, time;
int session_id = 5;

Serial.write(session_id);
 Serial.write(resultado);
 Serial.write(latitude);
 Serial.write(longitude);
 Serial.write(date);
 Serial.write(time);

Serial.write will not send the multibyte variable with that syntax. You need a byte cast.

Try

Serial.write((byte*)&resultado,sizeof(resultado));

You can do the same with a struct.

6

Try sending these example data from UNO to compatible receiver: Edit

int x1 = 0x1234;
long y1 = 0x7F1234AB;
long y2 = 0x12345678;

unsigned long z1 = 0xFE123456;
unsigned long z2 = 0x12345678;
unsigned long z3 = 0xCDAB1234;

The Sender Sketch:

#include<SoftwareSerial.h>
SoftwareSerial SUART(2, 3);  //SRX = DPin-2, STX = DPin-3

int x1 = 0x1234;
long y1 = 0x7F1234AB;
long y2 = 0x12345678;

unsigned long z1 = 0xFE123456;
unsigned long z2 = 0x12345678;
unsigned long z3 = 0xCDAB1234;

void setup() 
{
  Serial.begin(9600);
  SUART.begin(9600);
}

void loop() 
{
  SUART.print('<');  //START Mark of message
  SUART.print(x1);  //data
  SUART.print(',');  //data item seperator  
  //-------------------------------------- 
  SUART.print(y1);  //data item
  SUART.print(',');  //data item seperator 
  SUART.print(y2);
  SUART.print(',');   //data item seperator
  //--------------------------------------
  SUART.print(z1);
  SUART.print(',');  //data item seperator 
  SUART.print(z2);
  SUART.print(',');   //data item seperator
  SUART.print(z3);
  SUART.print(',');  //data item seperator 
  //----------------------------------------
  SUART.print('\n');  //Newline character (what is the use of it?)
  SUART.print('\r');  //Carriage return charcater  
  //----------------------------------------
  SUART.print('>');  //END Mark of message
  //-----------------------------------------
  delay(1000);       //test interval
}
7

Arduino Uno ?

Is the Arduino Uno connected to the USB bus of the Raspberry Pi ?
Do you use the serial-over-USB ? or have you connected the RX and TX pins between the Arduino and the Raspberry Pi.

The Arduino Uno is a 5V board. When the TX pin is HIGH, then it can output 20mA (or 40mA shortcut current).
The Raspberry Pi might not like that someone is putting 5V on one of its pins.

The Serial.write() writes one byte. Your 32-bits long values are converted to byte.

The Serial.write() can send more bytes when the amount of bytes is given.

Serial.write( (byte *) &date, 4);

When you have the Arduino connected to the USB of the Raspberry Pi and use serial-over-USB and you need to communicate between Python and Arduino, then there is a tutorial on this forum and a library on Github. I don't know which one on Github preferred by users of this forum. I can try to find that if you want.

Have you heard about the XY-problem ? http://xyproblem.info/.
When you want to communicate between Python and Arduino, then you were asking the wrong question and the fixed length is not important.

8

cattledog:
Serial.write will not send the multibyte variable with that syntax. You need a byte cast.

Try

Serial.write((byte*)&resultado,sizeof(resultado));

You can do the same with a struct.

This almost works! The only issue right now is how the additional zeroes are trailing the data instead of preceding it, or invert the order of the bytes so I can process it backwards on the other end.

I'll also look into what structs are and how they work, thank you!

Koepel:
Arduino Uno ?

Is the Arduino Uno connected to the USB bus of the Raspberry Pi ?
Do you use the serial-over-USB ? or have you connected the RX and TX pins between the Arduino and the Raspberry Pi.

It's connected through the GPIO pins of the Pi to the RX and TX pins of the arduino, Im waiting on some parts to arrive that would let me communicate them wirelessly which is my end goal, but since as far as I read the operation is going to be basically the same as far as software goes.
I did make sure the electrical levels were appropiate tho.

I need the binary packets to be of a fixed length to be able to decode them easily without relying on boundary characters, since transmitting the bytes of integers might cause it to fail if the value of any byte happens to be that boundary character.
And I can't use the Serial.print function since it would make the packet significantly larger, and since there will be duty cycle restrictions when transmitting wirelessly, I want to keep it as small and predictable as possible.

9

This almost works! The only issue right now is how the additional zeroes are trailing the data instead of preceding it, or invert the order of the bytes so I can process it backwards on the other end.

Please explain more about the data you are sending, the code which is receiving the data, what you are seeing and what you want to see.

10

You better stop what you are doing. Tell us which Arduino board you use.

I hope you are not trying to blow the pins of the Raspberry Pi.
When you plug the Arduino board in the USB of the Raspberry Pi, then you have already a serial connection.

11

Koepel:
You better stop what you are doing. Tell us which Arduino board you use.

I hope you are not trying to blow the pins of the Raspberry Pi.
When you plug the Arduino board in the USB of the Raspberry Pi, then you have already a serial connection.

I am using an arduino Uno board, and a raspberry pi zero. The uno is connected through the USB cable to a desktop computer, and through the GPIO pins and a voltage level adapter circuit to the pi zero. That part is sound.

cattledog:
Please explain more about the data you are sending, the code which is receiving the data, what you are seeing and what you want to see.

The part of the code in charge of the communication is now through software serial, since I will need to use both anyway in the future and the USB connection is being used by the desktop to check the data sent is correct.

GasMeter sensor;
TinyGPS gps;
/* rx, tx */
SoftwareSerial soft_serial(2,3);

long latitude, longitude;
unsigned long age, resultado, date, time;
int session_id = 155;
void setup() {
  sensor.initialize();
  soft_serial.begin(9600);
}

void loop() {
  while(Serial.available() !=0){
    Serial.read();
  }
  if( sensor.get_reading(resultado)){
    gps.get_position(latitude, longitude, age);
    gps.get_datetime(date, time, age);


    /*Testing output from the Arduino back to the computer to make sure it works*/
    Serial.print("[R]");
    Serial.println("Lectura del sensor");
    Serial.println(resultado);
    Serial.println("Latitud y longitud");
    Serial.println(latitude);
    Serial.println(longitude);
    Serial.println("Fecha y hora");
    Serial.println(date);
    Serial.println(time);

    soft_serial.print("[R]");
    soft_serial.write((byte*) &session_id, sizeof(session_id));
    soft_serial.write((byte*) &resultado, sizeof(resultado));
    soft_serial.write((byte*) &latitude, sizeof(latitude));
    soft_serial.write((byte*) &longitude, sizeof(longitude));
    soft_serial.write((byte*) &date, sizeof(date));
    soft_serial.write((byte*) &time, sizeof(time));
    delay(500);
  }

  delay(1000-(millis()%1000));
}

And on the raspberry I'm running this script

#! /usr/bin/python3

import serial, sys

arguments = sys.argv
arguments.pop(0)
serialPort = arguments[0]

ser = serial.Serial()
ser.baudrate = 9600
ser.port = serialPort
ser.bytesize = serial.EIGHTBITS
ser.parity = serial.PARITY_NONE
ser.stopbits = serial.STOPBITS_ONE

try:
    ser.open()
    while True:
        in_data = ser.read(3)
        if (in_data==b'[R]'):
            print(in_data.decode())

            session_id = ser.read(2);
            gas_received = ser.read(4);
            latitude = ser.read(4);
            longitude = ser.read(4);
            date = ser.read(4);
            time = ser.read(4);

            print("session_id")
            print(session_id.hex())
            print("gas_received")
            print(gas_received.hex())
            print("latitude")
            print(latitude.hex())
            print("longitude")
            print(longitude.hex())
            print("date")
            print(date.hex())
            print("time")
            print(time.hex())

except KeyboardInterrupt:
    ser.close()
    print("Closing")
    pass

As it is right now, this is an output

[R]
session_id
9b00
gas_received
c8af0000
latitude
88130000
longitude
88130000
date
bcc50000
time
e826e500

If you see the received session_id, for example, its 9b00, where the hex value for 155 is 009b. So I would need to put the 0s before the data.

12

Through a voltage level adapter, that is a big relief :smiley:
I don't know how Python reads data with ser.read(4) and how it can turn binairy data into a number. Sorry.
It seems that it reads characters. This is about reading binary data: Receiving binary data with Pyserial - Raspberry Pi Forums

13

Koepel:
Through a voltage level adapter, that is a big relief :smiley:
I don't know how Python reads data with ser.read(4) and how it can turn binairy data into a number. Sorry.
It seems that it reads characters. This is about reading binary data: Receiving binary data with Pyserial - Raspberry Pi Forums

The data is stored as a bytes variable. By using the bytes.decode() method you will get characters out of it, but there are other ways to turn them into integers, like the int.from_bytes() method.
The only issue is how the added zeroes are trailing and not leading

14

int.from_bytes() method.
The only issue is how the added zeroes are trailing and not leading

You can specify the endianness of int.from_bytes() to make the received bytes correct, or else bit shift them.

In the little endian world of Arduino the 2 byte integer 0x009b is stored in memory with the 9b at the lower number address and it is sent first. The rpi receives 9b and then 00. You should be able to simply recombine them into the 009b.
Same goes for the longs.

15

I still have a XY-Problem feeling :confused:
When you would send a single line with readable text and comma's between the numbers and a high baudrate, that would work. Then you can do a readline in Python and extract the numbers.

16

cattledog:
You can specify the endianness of int.from_bytes() to make the received bytes correct, or else bit shift them.

In the little endian world of Arduino the 2 byte integer 0x009b is stored in memory with the 9b at the lower number address and it is sent first. The rpi receives 9b and then 00. You should be able to simply recombine them into the 009b.
Same goes for the longs.

That's it! Turns out that was my mistake, believing my arduino was big endian instead of little endian.
Casting the integers as bytes, setting the fixed transmission size with serial.write, and converting them with int.from_bytes(ser.read(x),byteorder='little') gets me the correct results!