Arduino Acts weird when using servo libary?

Would it be better to manually write to the servo rather than use the servo library?

No.