I don't understand why the IR receiver would have a modulation frequency attribute. Is the TSOP4838's just a photo-transistor that tuned for IR wavelength?
A receiver like the TSOP4838 has a bit more magic than that going on inside it. It has automatic gain control; it works in both a dark room or full sunlight. It ignores lights flickering at different frequencies (e.g. fluorescent lights). And still on top of this it reliably detects your signal (your LED) which can either be quite bright or dim. Not at all a simple problem if you were starting with a run of the mill phototransistor.
When buying these IR receivers, why do they mention the modulation frequency at all? Isn't the modulation/demodulation done on the serial pins of the Arduino? My understanding is that the <IRremote.h> library is what is doing the demodulation, is that incorrect?
When the IR receiver (TSOP4838) sees six consecutive "blinks" of light at the 38KHz frequency it sets its output pin digital high and keeps it high until the moment that it calculates another blink should have occurred but didn't. The IR library then takes the lengths of those highs and the time between them and translates that into a code. So the IR receiver is expecting everything to occur at 1/38000ths of a second timing divisions, but the IR decoder library is only watching the lengths of the pulses coming from the IR receiver.
Do I need to buy a special IR transmitter that's also sold as "38kHz", or can I just use any old 940nm IR LED? The pictures of the IR transmitters on sparkfun definitely look like just regular old LED's.
You can use any old IR LED but the receiver is most
sensitive to IR LEDs of the 940nm wavelength. As a corollary to that it will still work with other LED colors but at shorter distances. I suggest you start with a white or red LED for testing so you can actually see the light.