EasyTransfer Library - sending and receiving array of bytes (or structs) from Python

Projects Interfacing w/ Software on the Computer
1

Hello everybody,
I'm Fabio, hardware developer from Italy. As this is my first message here I will present me as a novice C programmer (let's say that I can fill completely an ATMEGA328P without bootloader but still not so good when dealing with the dynamic part of the memory). I'm also a quite modest designer in the electronics, especially in the audio field and with mechanics.

So I'm asking your precious advices as I would give my contribute for the working environment of Mr. Bill Porter's EasyTransfer Library - being not so prompt to disturb him directly for this purpose.

These days I'm feeling particularly comfortable with this library for a project of mine involving (obviously) many ATMEGA328P linked in serial communication and sharing, one after the other, the same array of bytes - each MCU just receives the array, collect the data of its interest and foldback the same array for the next board. In order to achieve the fastest speed with many bytes on the tiny buffer memory of this MCU, I preferred EasyTransfer as this adopts a protocol of reasonable length.
As stated by Mr. Porter, the protocol is as follows:
Header(0x06,0x85), SizeofPayload, Payload, Checksum

I can say that everything works perfectly on the Arduino-to-Arduino side, but I'm finding the proper difficulty dealing with Phyton, as this is for me the first day of messing around with it.
Despite of SerialTransfer, which was developed also for communication with Phyton, EasyTransfer has no support on this side, so I'm sure that solving this point can be useful to many, and probably not only newbies like me in the matter.

The major issue I'm trying to work around can be found on the .cpp file of the library.
Mostly, the data to be sent, in the form seen above, is generated thru the following piece of code:

//Sends out struct in binary, with header, length info and checksum
void EasyTransfer::sendData(){
  uint8_t CS = size;
  _stream->write(0x06);
  _stream->write(0x85);
  _stream->write(size);
  for(int i = 0; i<size; i++){
    CS^=*(address+i);
    _stream->write(*(address+i));
  }
  _stream->write(CS);

}

In the specific example of my purpose:
my array of bytes has 8 elements and should something like:
[1,51,255,0,0,0,0,0] where the early four digits are the most important ones, the latters could be also discarded from transmission.
It's pretty clear that the header must be sent as 6 and 133 (in hex, 0x06 and 0x85), then I can see a 0x08 that should be the payload size (8), then some data that I can't distinguish, and a terminating checksum continuously changing.
In order to understand better what I'm receiving, some examples of the complete transmission are
x06\x85\x08\x013\x07\x00\x00\x00\x00\x00=\
or
x06\x85\x08\x013\x14\x00\x00\x00\x00\x00.\
or
x06\x85\x08\x013$\x00\x00\x00\x00\x00\x1e\

What can be the best way to generate, in the Python environment, a suitable array to be transmitted that can be read properly by the Arduino side of the EasyTransfer Library?
I'm still aware that getting back to SerialTransfer library is still a choice, but I've found it too much bulky (being casted for 16 bit elements and structs) and also a bit lacking in instructions: the data must be as light, fast and solid as possible and as yet said I feel that EasyTransfer fits perfectly for the job.

Thank you in advance for your contributions

EDIT: just forgot to say that the third value in my array is constantly changing between 0 and 255. Strangely, when I retrieve the readline from Python, I'm missing a value in that between:
x06\x85\x08\x013"\x00\x00\x00\x00\x00\x18
x06\x85\x08\x013#\x00\x00\x00\x00\x00\x19
x06\x85\x08\x013$\x00\x00\x00\x00\x00\x1e
x06\x85\x08\x013%\x00\x00\x00\x00\x00\x1f
x06\x85\x08\x013&\x00\x00\x00\x00\x00\x1c\
but it seems to be received pretty good from Arduino.
Is it supposed that the x013 (plus something) value is a sort of sum of at least two elements, if not three?

2

Actually I worked out the correct reading - it was tricky and not obvious.
By the suspicious misreading before discussed, I noticed that Python was receiving the serial data only when the third element in the array was constantly changing - in other words, setting it as fixed, or the whole array filled with the same value i.e. 1,1,1,1,1,1,1,1 was blocking any reading.
Then I added a "\n" at the end of any transmission, giving it as unnecessary.

Now the array is received complete with any value at its place from Python.

I still can't figure out how to send the data to Arduino, making it readable from the EasyTransfer algorithm.
My data in Python is sent this way:

value=bytearray([0x06,0x85,0x08,0x01,0x33,0xFF,0x00,0x00,0x00,0x00,0x00,0x08])
ser.write(value)

but I think something is missing, cause Arduino's RX LED blinks as something was received, but's still not properly reacting to the command.

3

Last time I checked the EasyTransfer library did not implement byte stuffing.
If that is still the case, then sometimes payload data can produce corrupt packets.
Check on this before issue before investing lots of time on a protocol that may not be suitable for your needs.

4

Yes! I still can't figure why, Python's readline(), apart of the previous correct reading, now misses again at least one element:
0x06, 0x85, 0x08, 0x13, 0xff, 0x01, 0x01, 0x01, 0x01, 0x01, 0xc4
and the array generated by Arduino is [1,51,255,1,1,1,1,1] , thus at least the first two elements were smashed.

The only thing is that the array is still transferred and processed correctly between the two Arduino's.
My true pain here is to understand how to produce in Python the right data packet that EasyTransfer expects for, on the board.

Thank you for your advice!

5

@mellaphon87 it seems you have everything right with the Python bytearray. This is a little test to echo back such a bytearray. I used your exact Python code


value=bytearray([0x06,0x85,0x08,0x01,0x33,0xFF,0x00,0x00,0x00,0x00,0x00,0x08])
ser.write(value)

Arduino

byte buffer[20];

void setup() {
Serial.begin(115200);
Serial.setTimeout(1);
}

void loop() {
while (Serial.available () > 0){
int bytesRead = Serial.readBytes(buffer,20);
Serial.write(buffer,bytesRead);

}
}

You could easily do a check for a header or checksum if needed

6

sumguy, the echo response to that bytearray is b'\x06\x85\x08\x013\xff\x00\x00\x00\x00\x00\x08'

7

Hi @mellaphon87 ,

here is a short sketch that does (almost the same as @sumguy 's :wink: but characterwise )

void setup() {
  Serial.begin(115200);
}

void loop() {
  if (Serial.available()) {
    char c = Serial.read();
    Serial.write(c);
  }
}

and a Python code that sends and receives the data:

import serial
import time

def millis():
    return int(time.time()*1000)

with serial.Serial('COM7',115200) as ser:
    outArray = bytearray([0x06,0x85,0x08,0x01,0x33,0xFF,0x00,0x00,0x00,0x00,0x00])
    checksum = outArray[2]
    for b in range(3,len(outArray)):
        checksum ^= outArray[b]
    outArray.extend(checksum.to_bytes(1,"little"))    
    inArray  = b''
    lastRead = 0
    lastSend = 0
    while True:
        if ser.in_waiting > 0:
            c = ser.read()
            lastRead = millis()
            if not (ord(c) in [10,13]):
                inArray += c
        if len(inArray) > 0 and (millis()-lastRead > 100):
                for b in inArray:
                    print(hex(b),end=' ')
                print()    
                inArray = b''    
        if millis()-lastSend > 2000:
            lastSend = millis()
            ser.write(outArray)

The Python script

  • sends a given byte array every 2 secs
  • receives the response byte per byte
  • and outputs the result in hex after a timeout of 100 msecs

In addition it calculates and adds a checksum byte to the end of the byte array (hopefully the same as your code from post #1 creates and sends).

If I understood it correctly this code

//Sends out struct in binary, with header, length info and checksum
void EasyTransfer::sendData(){
  uint8_t CS = size;
  _stream->write(0x06);
  _stream->write(0x85);
  _stream->write(size);
  for(int i = 0; i<size; i++){
    CS^=*(address+i);
    _stream->write(*(address+i));
  }
  _stream->write(CS);

}

transmits 0x06, 0x85 and the size first. Then a number of size bytes and finally the checksum CS. So in total this would be 3 plus size bytes. With size == 8 it would be 11 bytes, correct?

Therefore the Python script should also send and receive this number of bytes for a complete message. If "size" is not a fixed number it could synchronize on 0x06 and 0x85 and then read size for the rest plus 1 byte checksum.

Maybe I'm mistaken ...

Good luck!
ec2021

P.S.: I removed the last byte from the output array and replaced it by the checksum byte ...

1 Like
8

@mellaphon87 ok 11 bytes so I would guess removing the serial timeout would cure that?

I would rather leave a timeout in there just in case there is a problem with missing bytes the program could hang. So try extending it instead, maybe 10 .

You seem to be doing ok constructing the bytearray but often I might use a list just because its easier to clear, append, pop etc. Don't let me distract you.

BTW bytesRead will return the number of bytes in buffer so you can check packet length.

9

probably adopting a list is the best idea, but I will reach that point after solving this. I'm very new to python and of course I have to learn everything by now. I've done a lot of things today as the very first day but I'm stuck on this transmission. While awaiting for your response I was also attempting to make the same with SerialTransfer. No luck with the Python library pySerialTransfer as seems too much complex for my actual abilities - now the best I'm looking for is a simple statement to put under a callback_function() and still wrestling with the indentations here.
Further, the array is trasmitting up to four channels of AD conversion at very low sampling frequency and resolution for referencing purposes, so if some sample drop out is not a big issue, but EasyTransfer was accomplishing it very well with no drop-outs at 250000 baud.

Can't say you if timeout was critical as the data received (Arduino Serial Monitor) was some square symbol and then blank spaces, Python side was exactly the same than before.

10

Ec, the results of your sketch plus the python program gave:

0x6 0x85 0x8 0x1 0x33 0x25 0x0 0x0 0x0 0x0 0x0 0x1f
0x6 0x85 0x8 0x1 0x33 0x26 0x0 0x0 0x0 0x0 0x0 0x1c
0x6 0x85 0x8 0x1 0x33 0x27 0x0 0x0 0x0 0x0 0x0 0x1d
0x6 0x85 0x8 0x1 0x33 0x28 0x0 0x0 0x0 0x0 0x0 0x12
0x6 0x85 0x8 0x1 0x33 0x29 0x0 0x0 0x0 0x0 0x0 0x13
0x6 0x85 0x8 0x1 0x33 0x2a 0x0 0x0 0x0 0x0 0x0 0x10
0x6 0x85 0x8 0x1 0x33 0x2b 0x0 0x0 0x0 0x0 0x0 0x11

This is correct, we have 11 bytes if you don't consider the checksum one.
I've tried to send the array in this form from Python to the board with the following code

def callback_function():
ser.write(bytearray([0x6,0x85,0x8,0x1,0x33,0xff,0x0,0x0,0x0,0x0,0x0,0x8]))
c=ser.readline()
sg.popup(c)

that's written exactly as was received with your code but... no luck. The board receives it, but doesn't want to process the command I've sent.
Differently, works great if the code is sent via another arduino with the same EasyTransfer library... that's exactly the sequence written above! (where you can see the third element of the array fading between 0 and 255)

11

When my Python script sends the array with 0xff

  • 0x6 0x85 0x8 0x1 0x33 0xff 0x0 0x0 0x0 0x0 0x0 0xc5

the checksum is 0xc5.

In

  • ser.write(bytearray([0x6,0x85,0x8,0x1,0x33,0xff,0x0,0x0,0x0,0x0,0x0,0x8]))

the checksum (the last byte in the array) us set to 0x08.

Can it be that the EasyTransfer library discards messages with a wrong checksum?

What happens if you send the message as follows:

  • ser.write(bytearray([0x6,0x85,0x8,0x1,0x33,0xff,0x0,0x0,0x0,0x0,0x0,0xc5]))

Just a try ...

12

Solved! EC2021 thank you very much! Now it works like a charm.
I was completely not considerating that the checksum byte was calculated with some specific criteria that are quite new to me in this matter. I noticed this looking again to a new and longer sequence from the fading arduino.
As well as in your code (and in the linked from Mr. Porter too) there's a xor assignment operator, I've made the callback_function() following your algorythm:

outArray=(bytearray([0x6,0x85,0x8,0x1,0x33,0xff,0x0,0x0,0x0,0x0,0x0]))
    checksum=outArray[2]
    for b in range(3,len(outArray)):
        checksum ^= outArray[b]
    outArray.extend(checksum.to_bytes(1,"little"))
    ser.write(outArray)

this way letting me to transmit the 11 fixed bytes and then extending the array to the 12th strictly calculated from the whole set of bytes.
And now the transmission from Python works like a charm.
Still thank you and my best regards

13

That's great!

For a fixed command you can just add the correct checksum byte at the end of the array.

If you want to change the number of data you had

  • to use a byte array with only the first two bytes
  • then add the payload size,
  • then the payload byte by byte,
  • let the checksum calculate as above from byte no. 3 to the last byte
  • and finally add this checksum to the output array before you send it.

Glad I could help :wink:
Have fun!
ec2021

1 Like
14

Thus, the two headers are out from the computation of the checksum. I was wondering to discard also the payload size from the library, having a fixed number of data being transmitted for the whole project. The bytes in the array are always 8 so to avoid the continuous transmission of this datum and hoping to make things lighter, it should be given for granted by the library, at least in this case.
Mostly, I need to control (transmit) only the first four elements of the array, as the other four are just information collected from each MCU for analog/digital conversion purposes - the less significant task to do, actually.
Still thank you EC2021 and to sumguy and mikb55 for your precious help

2 Likes
15

Sorry for being unclear with "byte no. 3", I should have written index 3, Python (like C/C++) starts arrays with index 0... It's not only the first two signature bytes but also the size which are out of the checksum. The calculation starts with index 3 which is the forth byte in the array.

I inserted some comments in the code you posted to clarify how it is done there

//Sends out struct in binary, with header, length info and checksum
void EasyTransfer::sendData(){
  uint8_t CS = size;               // This line assigns the value of size to CS
  _stream->write(0x06);            // This sends 0x06
  _stream->write(0x85);            // This sends 0x85
  _stream->write(size);            // And this the size
  for(int i = 0; i<size; i++){     // After that(!) the calculation of CS starts!
    CS^=*(address+i);              // The value of CS is xored with each payload byte 
    _stream->write(*(address+i));  // which is send here
  }
  _stream->write(CS);              // Finally CS is send 

}

The Python line

  • for b in range(3,len(outArray)):

starts with b = 3 and ends with b = len(outArray)-1 as range(x,y) includes x but excludes y.

Maybe not relevant at the moment but I think it's good to know.

Regards to Italy (from Germany)
ec2021

16

Ec2021, thanks for making it so clear. I'm wondering how to adapt the source code from the library to the Python language, so new to me, comprising the formula to determine the checksum. Also thank you for the hint about the range - I noticed this when dealing with sliders, which range (0,255) was clearly excluding the last value and so I just added the 256 value in order to restabilish the full 8 bit control on it.
After my very first days of Python and my first 6620 lines of coding (that are running surprisingly fine also with your support) I think that's mostly intuitive, knowing a bit of C, but also many logical senses seem to be no to so obvious rather than the behavior of C/Arduino, but I'm going forth as much as I can by myself.
With your last code should be more easy excluding (or better, just making obvious) the size byte that, in my case, is fixed to 8. I'm saying it should because of course also the rest of the library must be corrected properly - not urgent now, but at some point, having the base of all coding just working, I will start to optimize details like this.
Still thanks! I should be touring Germany for holidays cause I love it!

17

I'm a bit late, but you may be interested in an easier approach via SerialTransfer.h and pySerialTransfer

18

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.