only thing i did was to start reading the serial port after 4 bytes.
Yep. That's the key. Don't try to read data that isn't there. Doing so will get you all out of sync.
The next step is to change the 4 values being sent to 6 values - including a start of packet marker and an end of packet marker. With these, you can tell if a packet is complete. A start of packet marker followed by 4 value that are not end of packet markers followed by an end of packet marker defines a good packet. Anything else, and that data just gets dumped.