Lets start with something simple. If i were to make a simple led circuit where if I pressed a button an led turned on.
So parts for the circuit would be:
1x button
1x led
Wire
1x 330ohm resistor
1x 1kohm resistor
Now if I wanted to make the arduino without using a pre-built, how simple can I make it? Would all it need is the atmega328, and the 5v voltage regulator?
Do you use a battery or usb power ?
I don't know an Arduino/Atmel chip that operates at 3 to 12V, so you are stuck with voltages between 2.7 and 5.5V.
Support you use 2 or 3 AA batteries, or a Li-ion cell or usb power. Then you can use a ATtiny85. It runs without external components and do so many things (too many things). But I prefer a 100nF decoupling capacitor at VCC and GND.
You can run the ATmega328P with its internal oscillator, it would only need a 100nF decoupling capacitor.
I have made a few barebones, but in the end it is always more work than I thought when I started. Connectors need to be added (to burn a bootloader and one more to upload a sketch), wires soldered, a voltage regulator, more sensors, add a I2C bus, and so on. In the end I should have use a normal Arduino board, but the project seems so simple when I started it 
You need only the IC plus power source ( 3x AA) 1 resistor and 1 LED.
Flatduino
Good design will include the reset pullup, reset button, and bypass cap for Vcc. But for blinky, you can wire Reset to Vcc and leave out the bypass cap.
The same goes for the tiny series, like the tiny85 1 second blink circuit:
1-second-tiny85-timer
Ray
Here is just one example