Do I need a mosfet driver?

I didn't quite understand the first part of your answer. You said my schematic will work, but then you said it will not work with 12V.

Just in case:
I am using 1S Lipo Battery (3.8V), 20khz pwm and the 8520 Brushed Motors.