Yeah, then the trouble is indeed that pin 4 isn't a PWM pin. Switch to pin pins 3, 9 10 and 11. Pins 5 and 6 have a different PWM frequency which is probably noticeable.
And you're missing enable for driver 2 
And if you want to do it all easier, switch to using the enable line as the PWM input to the driver and connect the A inputs to digital pins. That way you only need two PWM pins, one per motor.