weird problem with python code

I have this really weird problem with my python code and arduino
I have a simple motor controller setup on my arduino. I sent it serial data in three different packets and the arduino reads it and then intrepets which motor should move and how fast. Well everything works fine when I run the serial monitor with my python code. I first upload the firmware or reset the arduino then runing the serial monitor program from the arduino GUI. After that I have a python program that sends data serially to the arduino.

The problem happens when I don't have the serial monitor open. I can see the packets being sent via the lights on the arduino but nothing works. I don't see my motors move. Very weird but once I start the serial monitor everything works prefect.

Somethings I noticed. I don't do any Serial.prints on the arduino but only reads. So even if the serial monitor is on it won't print anything to my arduino GUI serial monitor screen.
If I do a time.sleep(1) after I open the serial port in python it sometimes works. sleeping for .05 works pretty good when sending data also.
When I send data to the arduino the led on pin 13 turns on first for a split second then the RX led comes on.

So right now I think the problem is that I am not giving it enough time to close and restart the serial port. Or I might be trying to write to many packets.

So I guess what should I do to make this more reliable/stable. What should my timings be before writes and maybe even possible reads on the arduino?

I can post code if you would like.