How to send and receive struct data packet over serial

yes it's critical. i was asked to measure performance frequently and it was a time consuming

i think the need to copy a potentially large block of memory for the reasons you suggested is unreasonable. i see no benefit to copy data to avoid a cast.

based on your suggestion, two copies may be necessary on the receive side: first to a generic structure where type, length and data are defined so that the type can be extracted and then to the specific struct (e.g. A_s) for that type.

i don't believe you have the low level experience i have where these issues are very important. much of my coding style/biases are learned from Unix and later Linux source. i guess i was fortunate the the low level code i worked on was in C and didn't have the "guardrails" you describe

while i've been around C++ since the beginning of my career in '85, i've never really heard a good thing about C++ except and complaints got worse. i believe properly written C++ can be as concise as C and i continue to look for that understanding, but haven't seen it (haven't met that guy)

i recently heard brian kernighan say he thought C++ had "guardrails" needed in very large applications (e.g. CAD) that C doesn't have. but i believe there are better alternative to C++ such a Go

arduino code hardly falls into the vary large application category. and there are other coding styles outside of C++ mandates that are used in very large application to minimize compilation times.

so while you made me more aware of the value of void*, i don't believe i understand (grok) the nuances of casting that you've pointed to.

In C, cast would allow you to do what you want and the compiler will oblige (and possibly you get a crash or bug); In C++ (and even more in other more strongly typed languages) there are guardrails to limit the risks and the compiler can enforce some of those.

I think Peter's point is just about what the C++ standard says and how to be sure to fall within the guidelines to be "future ready"

It's not because it's UB that it will crash or will stop working in the future. But it's a risk you need to be aware of.

There is no copy of the data. The memcpy is only there to make sure the optimizer doesn't mess up your invalid casts.

The aliasing rules and other restrictions imposed by the standard are there to allow the compiler to make certain assumptions about your code, which allows for better optimizations. If the optimizer can assume that two pointers refer to different objects (i.e. they do not alias), then there can be no dependencies between a write from one pointer and a read from the other. This allows for aggressive optimizations and vectorization.
For example, Fortran has even stricter aliasing rules than C and C++ (no two pointers can alias, even if they're the same type), and this allows for better optimization in some cases.

If your code violates the rules imposed by the standard, the optimizer's assumptions are wrong, and the generated code is (can be) incorrect.

For the Linux kernel, Linus simply said: “I will use GCC with the nonstandard flag -fno-strict-aliasing enabled.”
Now he's no longer bound by the rules of the C standard, only by the preconditions defined by GCC.
If you compile Linux with a compliant C compiler without the -fno-strict-aliasing flag, it'll break. See for example https://stackoverflow.com/a/2959468.

If you want your code to be correct, and without any undefined behavior-induced surprises, you have to follow the standard and use memcpy instead of invalid casts.
If you want to live dangerously and only want your code to work correctly using a single compiler with very specific flags, then you'll probably be fine if you know exactly what you're doing, and if you know exactly what the preconditions and assumptions of your specific compiler are.

This is not a difference between C and C++, the aliasing rules in C are similar to those in C++. A transcribed version of your initial example code with (Generic_s *) &a would not be valid C either. You cannot reinterpret one type as another in C.
The Linux kernel does it and gets away with it because it uses GCC-specific flags, but it's not valid C code.

See https://stackoverflow.com/a/99010 for a discussion of the strict aliasing rule in C (note that the union alternative is valid C11, but not valid C++).

You seem to misunderstand the goal of the strict aliasing rule: it is not at all a “guardrail”, it's an assumption that can be made by the optimizer.

What about Arduino to Arduino UART communication
I wrote my code line this

// transmitter
HardwareSerial unit1(1);
struct DataPacket
{
    int x;
    int  y;
}dp;
const unsigned long buffSize = sizeof(dp);
char uBuff[buffSize];
void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  unit1.begin(115200,SERIAL_8N1,17,16);
  dp.x=43;
  dp.y=21;

}
void loop() {
  sendDataPacket(&dp);
  //sendDataMemcpy();
}
void sendDataPacket(const DataPacket* dp)
{
  unit1.write((const char*)dp,buffSize);
}
void sendDataMemcpy()
{
  memcpy(uBuff,&dp,buffSize);
  while(unit1.available())
 {
  unit1.write(uBuff,buffSize);
 }
  
}
// receiver
HardwareSerial unit2(1);
struct DataPacket
{
  int x;
  int  y;
}dp;
const unsigned long buffSize = sizeof(dp);
char uBuff[buffSize];
void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  unit2.begin(115200,SERIAL_8N1,17,16);

}
void loop() {
  // put your main code here, to run repeatedly:
  

receiveData(&dp);
Serial.println(dp.x);
Serial.println(dp.y);

}
void receiveDataMemCpy()
{
  memcpy(&dp,uBuff,buffSize);
  while(unit2.available())
  {
    for(int i=0;i<buffSize;i++)
    {
      uBuff[i] = unit2.read();
      Serial.println(uBuff[i]);
    }
  }
}
bool receiveData(DataPacket* dp)
{
  // Function works. But with this stream so many garbage values are comming
  return(unit2.readBytes((char*)dp,buffSize) == buffSize);
}

when I transmit data sendDataPacket(&dp); and receive data receiveData(&dp); Serial.println(dp.x); Serial.println(dp.y);

I received original dp.x and dp.y which I have transmitted and with these values I received some garbage values.

If I use sendDataMemcpy() function how do I write my receive function?
Is there any Arduino lib available for serial data packet transferring ?

This copy to the buffer is unnecessary, it is fine to do unit1.write(reinterpret_cast<const char *>(dp), sizeof(*dp));.

This is most likely due to framing errors, which memcpy is not going to help you with. First of all you should check if there are enough bytes available to read before calling readBytes, otherwise it will time out after a second. Then you should probably look at the start and end markers as proposed by J-M-L, you can look at the RFC 1055 link I posted for inspiration.

I've never used it, but you could try the one posted by gfvalvo:

1 Like

your code gives me an error: invalid cast from type 'DataPacket' to type 'char*. Also I tried SerialTransfer uart_tx_data and uart_rx_data codes. It doesn't display values on serial terminal.

You have two different variables with the name dp, one is a pointer, the other is not. My code applies to pointers.

void sendDataPacket(const DataPacket* dp) {
  unit1.write(reinterpret_cast<const char *>(dp), sizeof(*dp));
}

there is no doubt that ignorant use of a re-casted pointer can result in undefined behavior or worse.

i thought Pieter was suggesting the casting itself was undefined. but there have been several misunderstandings in this discussion

i don't understand ... i allocate 2 buffers, i call memcpy to copy the contents from one to the other. i then operate on the 2nd buffer.

how can the code work if "There is no copy of the data"? ???

please see my comment to JML above.

i hope it's clear that i'm saying you need to know what you're doing when you recast a ptr. it's a tool for the developer; tools can be misused.

i think you're suggesting that i'm saying you can rely on the compiler to somehow solve coding problems. i'm not

There is no "ignorant use". If you cast a pointer to a different type (e.g. A_s * to Generic_s *) and use that invalid pointer, you're invoking undefined behavior, always.

The casting itself is not explicitly disallowed (though very bad practice). It's dereferencing the pointer that actually invokes undefined behavior.

If the memcpy replaces a reinterpret_cast, then the compiler will not allocate the second buffer and will not actually copy anything.

See the example I posted here:

The variable EthernetHeader hdr and the call to memcpy only exist in the source code, it doesn't actually get allocated in the actual binary, and there is no call memcpy instruction either.

This kind of optimizations is exactly the reason why you shouldn't violate rules like the strict aliasing rule: if you do, the optimizer has incorrect assumptions about your code and might optimize out the wrong variables/code.

That's not the point, what I'm saying is: you are never allowed to recast a pointer to a type that's not similar and use the result. Never. Not in C++, and not in C. Under no circumstances, regardless of whether you know what you're doing or not.

Sure, you can do it in assembly, which might be the cause for the confusion, I absolutely agree that when you look at it from a viewpoint of: "I have bytes in memory which I now want to interpret differently", it should work, but that's not the case when you program in C or C++. In those languages, you have to follow the rules set by the standard. If you don't, the compiler will mess up your code.

Although it's often useful to think about it this way, in C and C++, your memory is not just a large array of bytes. C and C++ programs operate on objects and values that are manipulated by an "abstract machine". The fact that objects have an object representation and that they're stored as a sequence of bytes is often a minor detail.
Certain stages in the optimizer reason about the interactions of objects in your code. If you mess around with the bytes in memory behind the back of the optimizer, if you access memory through aliased pointers or if you have pointers to nonexisting objects, the optimizer cannot produce the correct output.

It's not about the compiler solving code problems. It is about code that can be compiled correctly.

When you use a compiler, you enter a contract: you give the compiler valid C or C++ source code that adheres to the standard, and the compiler gives you a compiled binary that has the behavior required by the standard for the given source code.
If you fail to meet your requirements, e.g. by doing invalid reinterpret casts or by violating the rules set by the standard in other ways, the compiler does not have to generate any meaningful output.

Because of the way compilers are implemented, you often get something that happens to work, but there are no guarantees whatsoever, and it'll often break when you increase the optimization level or when you upgrade to a smarter compiler.

The following is a real-world bug that I've come across:

struct Vec2f {
  float x, y;
};
void normalize(Vec2f *v) {
  float norm = std::hypot(v->x, v->y);
  v->x /= norm;
  v->y /= norm; 
}
// in a different part of the code base:
struct Point {
  float x, y;
};
void normalize(Point *p) {
  normalize((Vec2f *)p); // this is not allowed
}

This worked perfectly fine, until one day some logging code was added to a function that used the normalize(Point *) function: the optimizer decided to take a different path, and the "normalized" point contained complete garbage.
It worked perfectly fine for years before that, and if you subscribe to the (incorrect) "everything is an array of bytes" view, it should work fine, but the truth is that it is not allowed by the standard, and the compiler does not have to generate valid code for it.

If you're not convinced by this example, see the link I posted earlier (https://stackoverflow.com/a/2959468), where the compiler reorders code because of an invalid reinterpret cast in the memcpy implementation. As long as you use the nonstandard -fno-strict-aliasing flag, the binary does the correct thing. If you use standard C, the optimizer has incorrect assumptions, the optimizer reorders certain instructions and the resulting binary gives bogus results.

@PieterP

I think the question was : if you have a structure you need to fill in from a byte stream how do you "legally" avoid having a byte buffer that you later on memcpy() into the structure and can you use the structure's memory directly to receive the byte stream.

I believe what I posted in Reply #47 covers this.

so in a nutshell this is legit... (I inlined what could be in a function)

struct __attribute__((packed)) t_buffer {
  char s1[2];
  char s2[2];
};

t_buffer myBuffer;

void setup() {
  Serial.begin(115200);
  char *ptr = reinterpret_cast<char *>(&myBuffer);
  *ptr       = 'A'; // assume 'A' comes from a byte stream
  *(ptr + 2) = 'B'; // assume 'B' comes from a byte stream
  Serial.println(myBuffer.s1); // (the trailing null char is there because of the global nature of the buffer)
  Serial.println(myBuffer.s2);
}

void loop() {}
1 Like

consider how you would process an application packet that can be of multiple types

void
processA (
    A_s *p )
{
};

// -------------------------------------
void
processB (
    B_s *p )
{
};

// -------------------------------------
int
receive (
    void * buf )
{
    return 0;
};

// -------------------------------------
void
processPacket (
    const void *buf )
{
    Generic_s   pkt;

    int nByte = receive (& pkt);

    switch (pkt.type)  {
    case TYPE_A:
        processA ((A_s*) &pkt);
        break;

    case TYPE_B:
        processB ((B_s*) &pkt);
        break;
    }
}

the question is do you need to use memcpy() to move the buffer back into a structure or can you use the structure's memory as a byte buffer. see my example code (which I think is in UB world)

You can allocate the structure, reinterpret it as an array of bytes, and fill the array.
This is allowed by the strict aliasing rule because of the exceptions for char, unsigned char and byte.

What you cannot do is allocate an array of bytes, fill it, and reinterpret it as a structure.
This is because an array of bytes is not (and is not similar to) your structure, performing such a cast would violate the strict aliasing rule.

If you have an array of bytes and you want to "reinterpret" it as a structure, the only valid way I'm aware of is using memcpy. As shown in my experiment above, the compiler can optimize away the memcpy, it aliases the array of bytes as the struct for you, but this is not something you, the programmer, are allowed to do yourself using reinterpret_cast, you have to rely on the compiler.

I think so, as long as t_buffer is TriviallyCopyable (see below).

For the reasons discussed above, that code is not valid. You'll either have to use memcpy and hope that the compiler optimizes it away, or use nonstandard GCC flags.
I agree that this is a less-than-ideal situation, but you should explain that to the C/C++ standards committees, don't shoot the messenger.

If you know any other valid alternatives, I'd love to know!

The description on C++ named requirements: TriviallyCopyable - cppreference.com isn't entirely clear, I'd have to check the standard to be sure but I don't have the time right now.

In general, for any trivially copyable type T and an object obj1 of T, the underlying bytes of obj1 can be copied into an array of char, or unsigned char, or std::byte (since C++17) or into obj2, a distinct object of T. Neither obj1 nor obj2 may be a potentially-overlapping subobject.

If the underlying bytes of obj1 is copied into such an array, and then the resulting content is copied back into obj1, obj1 will hold its original value. If the underlying bytes of obj1 are copied into obj2, obj2 will hold obj1's value.

Underlying bytes can be copied by std::memcpy or std::memmove, as long as no living volatile object is accessed.

It's unclear to me whether the last sentence implies that you can only use memcpy/memmove to copy the underlying bytes, or if you're allowed to read/write those bytes yourself. Reading/writing those bytes directly doesn't violate the strict aliasing rule, but there might be other nuanced restrictions.

why?

Strictly speaking the code is not invalid because it doesn't do anything useful.
However, if you access any of the members of A_s or B_s in the processA() or processB() functions, you are violating the strict aliasing rule: an object of type Generic_s is not an object of type A_s and the types are not similar, so you are not allowed to access it through a pointer of type A_s *. Why? Because the standard doesn't allow it.

See the explanation here:

so I think that would answer questions raised in 53/54

apparently gcc is unaware of "the standard".

i understand that behavior is undefined when using a ptr assigned a address of different type than the ptr due to casting. i can work with that

i'm fine with that and i believe most professional development groups are as well.

It is absolutely not. I'm not sure why you quoted “the standard”, C and C++ have internationally agreed upon ISO standards. They specify exactly what is a valid C or C++ program and what is not, and they specify how compilers should interpret your source code. Of course GCC is aware of the standard.

I have a hard time believing that. I've shown you multiple real-world scenarios where GCC actually produces invalid code because of such violations.

You seem to misunderstand the definition of Undefined Behavior: It doesn't just mean that the behavior is not defined by the standard (that would be unspecified or implementation-defined behavior), undefined behavior means that your compiler isn't required to produce any useful output, and the generated code could do anything.
You absolutely do not want this in your code. You can do everything else right, have correctness proofs for every algorithm in your code base, have all your unit tests pass, if you invoke undefined behavior in your code, all bets are off and the entire program is fundamentally broken.

If you don't care about any of that, that's your decision, but please don't teach these practices to others.