Electrolytic - Big values (usually 0.1uF and up to multiple F) in a relatively small space, but horrible at everything else. Typically 5-20% tolerance. Used for anything where large values trump all else, like rectified AC-to-DC PSUs to remove AC ripple, or to store a reserve of power to negate the effects of sudden current draw, PCB inductance, minor continuity issues, etc. They're polarized unless you get bipolar, which is just two polarized stuck back-to-back. Voltages usually range from 5v to 100v. Typical through-hole form factor is radial: a can with legs on the bottom, or screw terminals for really big ones. Electrolytics can be "slow" due to ESL -- equivalent series inductance -- inherent to the package, so a low-value ceramic is often used locally to shunt HF noise. They also "age" more than other types, and will dry out and lose capacitance over time -- especially if they're not frequently used. Cheap as dirt.
Ceramic - Medium values (pF to a few uF) in tolerances of 5-20%, used most often to couple PSU noise to ground due to good ESR/ESL performance. Using a 0.1uF (100nF) ceramic in parallel with a 10-100uF electroyltic is an easy way to keep a nice steady reserve of noise-free power at the Vcc pin of an IC. Small values (10s of pF) are also commonly used for quartz crystal load caps, and RF filters at analog or power inputs. Ceramics are not polarized, and typically come in voltages of 25v to 250v or more. Can be either axial (lead at each end) or radial in through-hole form, usually radial with blue or tan colored disc-like head. Large ones used for AC PSU RF filtering and switch arc prevention are larger, have special safety ratings to fail open, and come in yellow, tan, or black. Cheap for reasonable values.
Film - Small values (pF to nF) with excellent tolerance (up to 0.1%, but normally 1% to 10%) used when performance is crucial. Often used for audio and video circuits, for filters (low/high/bandpass), and the like. Very good ESL/ESR figures. Not polarized. Physically, film caps are extremely large for their values. Typical through-hole forms are either radial blobs in a tan, reddish, or brownish color, or radial rectangular boxes of various shapes and sizes. Relatively pricey, and some niche caps can get ridiculously expensive.
Tantalum - Similar to ceramic, AFAIK. I don't actually know much about tantalum caps, since I've never used them myself (to my knowledge). I think they're polarized, but maybe not always. Someone will need to correct me if I'm wrong there.
