so should I calculate resistor for 5V or for 5V-1V=4V (LED current is 320mA)?
Yes you should include it, however you also must include the Vf drop of the specific led you are using.
For your example lets assume a 3vdc Vf drop for the led assembly as I don't know the spec for your led.
R = (5v - (1v + 3v)) / .320 = 3.1 ohms
By the way, those high current led assemblies almost always recommend not to try and use a simple series current limiting resistor to maintain proper current control. They demand a constant current driver to operate safely.