Yes, it normally only occurs during the initial press. You may also see bouncing when releasing the button.
If it's a button you have to hold, that can get tiring and the person may release the button. You can add an extra "debounce" if you think that someone releasing the button and then pushing it again less than half a second later should be ignored. This is not normally a good design.