I think the max gs voltage is too high on the mosfet I choose. I need this to have a max of 1.7V and my mosfet has 2-4 volt value. More info about this is here: http://ics.nxp.com/support/documents/interface/pdf/an97055.pdf
Time to order a new mosfet and see if that fixes the problem...