I had the exact same question, i all got answered in my Computer Architecture class in university.
The atmel avr chip that is used in the arduino contains on integrated ressonator and some circuits that can put one cristal oscilating and producing one nice square wave that supply the chip internals with a stable known clock, the arduino uses the second.
The program that you create in the Arduino IDE is stored in one Flash memory, pretty much like your usb pen, so like your pen it doesnt "forget" what was inside it when you remove it from the usb(aka remove power).
When the avr chip gets a nice and stable source of power it runs and hardware init sequence that puts the clock/ressonator runing and waits some time so the square wave can stabilize and thus become usable.
This square wave is used to drive a lot of peripherals, just to name a few: clock source for timers, for the flash memory, for the adc circuitry and for the core itself.
In the core there is special register called program counter, this register as always a start up value of 0, the avr core grabs this value and use it as an address to where to read the flash memory, 0 being the very first instruction(as said above in the chip level its just 0 and 1's, but think of it as assembly), then each instruction can do things like saying to the program counter to jump to a specific value, this would be the result of one if or case comparation or a function call/return in C/C++ or some of the multiple types of branches or jump's that you can do in assembly, if it isnt a jump instruction then the program counter just gets increased by one, the new address is read from the flash memory, the new data read from the flash goes through some circuits that can decode it and say what instruction is and what and where things will go, now it can be and add instruction, so there must be two/three registers involved, and the ALU(google about this).
It isnt very complex but it is extremely fun being able to really understand how a CPU works.
Hope you understand this 