If you really want to do things right on the resistors, go with an EIA decade series:
Probably the easiest for the hobbyist is an E-24 series; this will give you a wide range of values and won't require the huge storage investment an E-48 or 96 series would require. Don't try to cheap out and go E-12 with 5% tolerance parts; you'll end up with "holes" in the values (because the tolerance won't cover the "gaps").
As far as caps, that will more depend on what you end up doing than anything else. You're on track with the 0.1uF caps; those are mainly used for filtering on digital ICs (connected between ground and power).
If you are planning on using ATMega8/168/328 microcontroller chips only (for standalone Arduino implementations, perhaps), and you plan on using crystals (as opposed to resonators), you will want to find some 18-33pF caps for the load capacitance (22pF caps are "perfect") of the 16-20 MHz crystals.
You will also want 16 (and maybe 20 MHz) crystals - or you may go with resonators (instead of crystals), and forgo the load caps.
If you are planning on working with any voltage regulators (again, for standalone Arduinos); get some 7805s - and get some 47uF electrolytic caps, plus some 1N4001 or 1N4004 rectifier diodes.
If you plan on working with higher voltages that you need to regulate, say 12 volts DC, then get some 7812 voltage regulators as well.
You might want an assortment of NPN and PNP transistors, in TO-92 cases (small signal transistors); if you are going to play with homemade h-bridges for larger motors, then such transistors in TO-220 cases are better (or for really large motors, TO-3 cases); note that you will also want heatsinks and mounting kits for these too. Or you might go with MOSFETs - really, you might be better off buying these items as-needed in a project, unless you can find good deals on them.
L298 motor drivers are good to have on hand as well - also ULN2803 and ULN2003 can be handy for many thing. 74HC595 shift registers are also a popular to have on hand Arduino item...
Small motors, LEDs (red, green, yellow are cheapest - leave the RGB, Blues, and Whites for special projects). Depending on what you are doing, some microswitches, potentiometers, a few (or many) servos, some relays, etc - all very useful to have on hand.
To determine whether you need or want a lot of this stuff, though, will be mostly determined by what you plan on doing; if you were planning on building visual displays or light-based art - will require a different kit than say, working with sound synthesis, which is different than say robotics...
Good luck - hope this helps and/or inspires!
