How to calculate if I am within limits?
Add up all the current you are expecting the Arduino to source or sink. The sum has to be less than the limit.
Does changing resistors will change it?
Yes it will change the current taken by the LED and so change the sum total of current.
Should I connect power pins of my five SN74HC595N registers to Nano's VIN insted of 3.3v?
The power pins should have the same voltage as the signal voltage, which is the power given to the Nano's chip.
Also each shift register should have a 0.1uF ceramic capacitor between the power and ground pins.