Looking at the website of the chip's manufacturer
, I see that the max supply voltage is 5.5V with an absolute max rating of 6V, while I understand from your initial post that you are supplying it with 12V. Perhaps that explains why the circuit is not functioning as desired?
The chip may be dead as a dodo. Your arduino may not be too happy if it's getting 12V on any pins except the one for raw power to the voltage regulator.
As for selecting the resistor Rset between the power supply and the Iset pin, the same page
seems to recommend a typical value of 9.53K for a current limit of 40mA sourced from your segments drivers, while the datasheet
seems to recommend the same value as a minimum, and states that the current limit is 100 times the current flowing into the Iset pin.
I would assume that a resistor twice the value would create a current half the intensity and recommend to respect the minimum value for Rset. As for calculating Rset for a supply voltage of 12 volts, if that was part of your question, I would contend that the question is moot since at 12 V the IC should not be able to function.