The gate itself is already a capacitor (sort off).
Adding 10-100n between gate and source might be enough.
Diode leakage might be your enemy. I would try the 1N4148 instead of a schottky.
But, the piezo is also a capacitor (sort off).
Could be that the piezo is generating too much voltage when hit.
Make sure the gate stays below the abslute max GS voltage (with 12-18volt zener diode?).
Post a proposed diagram before going ahead.
Leo..