If you go for lower load you will get more pullability and thus less precision / stability.
Yes, you are correct as the pullability is proportionally to the inverse of load capacitance square. I was also thinking of removing the 2 capacitors (that's why I was saying about 6pF crystal, :)), but obviously it will increase the ppm value too.