I prefer (although others may not agree 
) Sparkfun for arduino-related orders:
And also adafruit:
Now, depending on the setup you will have, you'll need to choose the appropriate arduino variation:
But you should start with the Uno (the basic board to get started with), preferably in a "starter's kit", that way you can first get the feel for the IDE, the language, and the board with some practice tutorials and a bunch of components in the kit:
^ this kit should get you started well.
For the lcd, a simple one would be
but if you want to be more fancy, you can use any on this page:
(the graphic ones would let you draw some more complicated stuff, and of course give you access to a relatively large array of pixels, whereas the first lcd is a simple character-style 16 columns by 2 rows setup, so you have access to 32 characters to display)