send and receive HEX over TTL

I do get some feedback over USB, but I don't know if it works over Tx Rx as well.

Which Arduino do you have? For most, "over USB" and "over Tx Rx" mean exactly the same thing.

Since I always get back some characters instead of hex code it does make it a bit weird to me.

The code and the serial output? Though, if you are talking to the lawn mower using the serial port, you can't also talk to the PC, so I see no basis for this statement.

Should I use if(Serial.available() >= 5) in the loop or is it ok to use it directly after sending?

Yes. Think about that statement, though. Sending serial data is like typing in this text window, and the hitting Post.

Can you reasonably expect the reply to be there immediately?

I'm afraid of having a timeout because the Automower is too slow in response.

Why?

So maybe including a delay before reading would work as well, since the Arduino has a buffer for serial communication.

"would work" and "would work as well" are not the same thing.