There is still a delay(20); in there.
delay(20);
Better use a debounce library like vaj4088 mentionned.
Jacques