First off, for a lot of components, etc - Ebay can be a great place to stock up on parts. Next would be surplus outlets like All Electronics, Electronic Goldmine, Alltronics, etc. Use places like Digikey and Mouser for items which are "new only" or you can't get any other way. New parts aren't cheap, but sometimes that's the only way to get certain components.
Regarding learning "Ohm's Law" and such - I can recommend two great sources of information:
1) Grob's "Basic Electronics"
2) Forrest M. Mims III "Engineer's Mini-Notebooks" - http://www.forrestmims.org/
The first is a textbook - essentially the book for EE101-type courses; as such, a brand-new, current edition is going to run you about $100.00+ USD; instead, pick up a used copy, something recent (within the past 10 years will do). As a text book, it covers a ton of material, and is heavy on the math and theory; it starts out with "What is an electron" and works up from there. You will get math, theory, and practical application, all in one book. Entire chapters on what resistors, capacitors, transistors, etc are, how they work, how they are made, the math behind them, theory of circuit operation, how to practically apply them in circuits, etc. This is a book that should be on your bookshelf.
The "Engineer's Mini-Notebooks" were originally published by Radio Shack back in the 1970s-1980s - but the originals are "out of print"; new editions do exist, and tend to cram more into a thicker and slightly larger booklet than the originals. They cover a ton of topics; everything from beginner materials to more advanced circuits. This is a great series to have.
One other source might be to look into old TAB Books publications - back in the 1970s and 80s (and a bit into the 1990s, until they were bought by another publisher), they published a ton of books on a variety of topics (not just electronics). They all had the same formats and layouts; once you read one TAB book, you know how the others will look and read. They had quite a few volumes on various personal hobby robotics projects and general information (How to Build Your Own Working Robot Pet by Frank DaCosta, for instance), many on computer programming of the era (lots of BASIC, and 6502, 8088, 6809 assembler, for starters), and several on building your own computer (based on now-old chipsets and CPUs, like the aforementioned ones). Excellent sources of information, when you can find them.
While none of the above publications will answer questions you might have about the Arduino, they should be considered an essential part of your bookshelf for electronics...
Hope this helps!