Think it goes wrong in readBytes().
What does this print? (I expect a value less than packet size = 4)
Analysis
If there are 3 bytes available of the 4 to receive, these will be copied in buffer by readBytes and in the next Loop these 3 bytes are ignored. So good bytes are dropped because the packet is not complete.
Better is to wait until at least 4 bytes are in the buffer (still not fool proof, just a bit better)
void loop() {
if (rs485.available() >= 4) {
uint8_t buffer[128];
size_t len = rs485.readBytes(buffer, 4);
Serial.print("SLAVE <- Otrzymano: ");
printHex(buffer, len);
A robust solution should do something like
- read one byte at the time
- check if this is a PACKET_START byte (define a start of the packet byte)
- if so collect the next N bytes as payload and CRC (one byte at the time)
- check CRC
- process data