Where is "here"? Knowing where you are in the world can help us help you find suppliers that are closer.
If you are in the United States, you would do well to look at those beginner kits already mentioned from Sparkfun and AdaFruit, and build up a list of components. Then, take that list and price out the items from Ebay, as well as from your local shops. Also - check out online places like Servo City (for servos), and surplus electronics places like All Electronics (http://www.allelectronics.com/), Electronic Goldmine (http://www.goldmine-elec.com/), and Alltronics (http://www.alltronics.com/) - I mention these because I have ordered from them in the past, and have had great results - YMMV.
Note that via Electronic Goldmine, they offer these things called "Surprise Boxes" - basically grab-bags (well, boxes) of various electronic parts and stuff (I call 'em "floor sweepings" - but the boxes I have ordered always had great stuff in them; well worth what I paid) - here's their "Super Surprise Box", the largest one they sell (AFAIK):
http://www.goldmine-elec-products.com/prodinfo.asp?number=G9321
Anyhow, with proper shopping - you can generally get together all the parts you need (and then some) for much less than those kits will cost you; you might have to do some sorting in the end (and with the "surprise boxes" from Electronic Goldmine, you'll be sorting for days) - but it is well worth it. Figure out a storage method early; it's important to be organised in this hobby.
Also - you didn't note it, but if you don't have at least a multimeter - GET ONE. You NEED it; it is really a necessary hardware debugging tool. Even if all you can afford is an el-cheapo Cen-Tech chinese version from Harbor Freight, it is better than not having one at all.