HARDWARE -
What products would I need for a simple button and microphone?
You might want to get a "Arduino starter kit" but I can't recommend one so maybe someone else can help.
The "extra" parts in the starter kit will allow you to try some of the other example programs so you can learn some more programming concepts. i.e. You might use some ideas from an example in a different way with your button & speaker.
Not a microphone. A microphone converts acoustic sound to an electrical signal. You want the opposite - a speaker (or piezo "transducer") to convert electrical signals to sound.
The Arduino can't drive a regular 4 or 8-Ohm speaker so I'd start out with a piezo transducer and then if it's not adequate you can add an amplifier and speaker. (Or, you can use "powered" computer speakers.)
There are two kinds of piezo devices for generating sound. Like I said, a "piezo transducer" or "piezo speaker" converts electrical signals to sound and that's what you want if you want to "make music", etc.
There are also "piezo buzzers" or "beepers". These have a built-in tone-generating circuit so you can't control the sound. It gets tricky because some people (and some "cheap" suppliers) say buzzer or beeper when they really mean transducer.
For the button, you generally want a "momentary pushbutton switch".
SOFTWARE -
There are [u]Examples[/u] for how to read a pushbutton switch and for making tones, etc., and each example has a schematic showing the required electrical components and wiring.
The two most important concepts in programming are conditional execution (if statements, etc.) and loops (doing something over-and-over, usually until some condition is reached). Once you understand those concepts you can start writing "useful" software.
When you start writing your own programs, don't try to write the whole program at once. Write and test one or two lines of code at a time. That's not as simple as it seems because the compiler has to "see" a "complete program", so you can't just start at the top and work down. (i.e. If you delete the bottom half of a working program, it's not going to compile.)
For testing & debugging, it's helpful to use the serial monitor to display variable values and to send little messages about what your program is doing. (See the Analog Read Serial Example.)
For your "user interface", in some cases if you can use toggle switches or potentiometers the user can see the physical setting/position. Other times a few LEDs can show "status" or a certain "beep code" might tell you what's going on, etc. Or, you may need an LCD display and keypad or multiple buttons, etc.