gkzsolt:
In order to connect some other, power hungry peripherals I would connect them to another power supply, sharing only the GND and control pins with the Arduino. (I understand that this could be a bit difficult if the shield is connecting its power to the Arduino)
Apologies for the tardy reply. I have been distracted by work.
The bread board PSU is two voltage regulators and some support circuitry. You can plug up to 12V into the power jack and what comes out of the pins is 5V and 3.3V. I am not using the bread board PSU's USB connector, because I am not certain which side of the regulator it is connected to. The documentation I found, says it is for powering a USB hub, which would make it an output.
If you are using the bread board PSU to power some power hungry peripheral. You can hook a wire from the bread board +5V rail to the VIN rail on the Arduino, and power both devices from a single PSU (wall wart, USB charger Etc).
I am using the USB as well for the moment..
Well, that's not the same circuit 
My board works with USB power, should it not come from my power board USB connector (mentioned in the first post.) But with an external USB charger it works.
I may use one or the other, or both at the same time.
If you have a multimeter available, see what happens when you connect a couple AA batteries
i) Positive to negative, in series - the Voltage doubles.
ii) Positive to positive, negative to negative, in parallel - the Voltage remains the same.
When I connect power to both the Arduino USB and the VIN pin, it is similar to connecting two batteries in parallel.
So you use a Vin + USB power? That's interesting, and would be interesting to know how the power is split up between the two (if that's the correct English terminology - sorry for it). Anyway, I would choose another solution.
Have a google for Ohm's law, and Kirchoff's laws of Voltage and Current.
The technical term for 'split' is 'divide' E.g. Voltage divider.
5V pin. Is a positive output. [...] One volt being much like any other volt, you can probably connect your own 5V regulated supply to this pin without causing damage but any more than 5V is likely to start frying components upstream.
Oops, I would never do that! Never connect 2 output power supplies together, only if you are sure what you are doing.
I know roughly what I am doing
Have a read of Ohm's law and Kirchoff's laws of Current and Voltage. The 5V pin is the output of the Arduino's onboard regulator. Kirchoff states that it is relatively safe to connect the 5V from a different regulator. Ohm's law states it is wise to fit a current limiting resistor, to prevent the Arduino drawing more current than it can tolerate.
But you are right. You should use the pins as the designer intended if there is any doubt.
IOREF. Is a positive input to the ADC.
That's a bit a source of confusion to me.
There are a couple reasons you might want to use the IOREF. The easiest for me to explain, is using an analog sensor with an output Voltage < 5V. For example, a thermistor circuit which measures temperature from 0C to 25C, with each degreee C equal to 250mV. The maximum output voltage the sensor can produce is 2.5V. Were you to connect just the sensor output to an analog pin, the Arduino would divide the Voltage on the analog pin, by the internal 5V supply and analogRead() would return values between 0 and 512. By connecting 2.5V to the IOREF, the Arduino divides by 2.5V and produces values between 0 and 1023, doubling the apparent resolution of the sensor.