That's not called a dimmer, that's called a variable voltage power supply or voltage regulator.
Normally, with leds, that's done by varying the duty cycle instead of the voltage, using PWM.
Some time ago I installed LED strips on my stairs. The strips I purchased had a high LED count which I wanted so the "dots" could be minimized. Because of this they were very bright, being my first LED strip project I though I could just drop the voltage and all would be good. Such was not the case. At very low brightness levels the individual LED's operated differently giving me bright and dark LEDs on the same strip. To make them all the same brightness I had to pulse them on and off quickly (called PWM, you can google it).
Now my brightness reduction was an extreme case your dim goal might be different from mine. But you should be aware using a resistor to drop the voltage can cause this issue.
You should look into PWM if not for this project, its good to know for the future. Also it will make your batteries last much longer.
However if you really want to use a resistor you need to do/know the following:
Pots available today are almost all low power, not capable of dimming a typical LED strip.
You need to know how much current the LED strip draws at you lowest dim point and the highest brightness (likely direct connection with no resistor). The resistor you will need for the dim operating point is: R = voltageDropNeeded/current
The resistor will dissipate power at the dim point is: current * current * resistance. You should use a resistor with 2 to 3 times this power rating.
if You use a pot of any rating, that rating only applies to the whole of the track. It the pot is being used as a variable resistor as opposed to a variable potential divider then as you get less resistance more current gets dissipated over an increasingly smaller section of the track. This will almost certainly burn out the track unless it is massively over specified, like the old fashioned spot light controls, where the dimmers were two feet long or more.