Yes, the program (aka sketch) is remembered in the internal flash memory of the device. You also have EEPROM memory in the device where you can store device specifics (such as a unique ID) and other configuration data that will be remembered when power is removed. Be careful with this memory as it has a limited amount of writes - around 10,000 (or is it 100,000?). Then there's RAM which is used for your general program variables etc - which gets lost when power is removed.
Do you have a controller device in mind to oversee all of this?