But if I half the voltage, doubling the current should only increase torque, not speed correct?
That is a correct statement. Dropping the voltage will reduce the speed, not the available torque. In order to maintain the same power output at half speed one must double the torque. This may or may not be possible due to 1) the load, 2) the motor and 3) the power source. Those three issues, one at a time:
The load type connected to the shaft will determine if the torque required will change with speed. I don't know what the OP's load is so here are the three basic load types.

Constant torque A load where the torque required is independent of the speed applied. A material conveyor or screw feeder are examples. The power required is linearly proportional to speed since torque remains constant.

Variable torque Sometimes referred to as Quadratic torque. The most common example would be a fan or propeller and most pumps. The torque required to turn a variable torque load goes up at a rate of the speed squared. Since power is speed times torque, the power required to turn such a load goes up at a rate of the speed cubed. This has a huge effect on the torque required and good example is this: For a given system at 100% power and 100% speed, reducing the speed by 50% would reduce the power by 87.5%
With a variable torque load, reducing speed will always result in a significant reduction in actual torque, you cannot increase the torque required when decreasing speed.

Constant power The required torque is inversely proportional to the speed. An example would be a grinding wheel.
The problem with small hobby motors is that the specifications generally do not provide a maximum power rating. Voltage and torque ratings are given but good luck finding useful data like torque per amp/milliamp, you have to extrapolate that from the stall torque and current. Without a power rating, it is difficult to predict what the motor life will be for a given set of conditions.
Small hobby PM motors are high speed with corresponding high frictional losses  which cause most users to misinterpret what they see with regard to the three basic physical relationships that exist (stated in post #2) in a PM DC motor. Ultimately, is the motor selected capable of running at the required torque? How far away from the stall torque is that rating? I'd venture a guess and say anything more than 50% of stall may lead to shortened motor life. I'm not knowledgable there so I leave that to others to comment.
Lastly, there is the issue of available current. Can your power supply provide what the motor requires to produce the desired torque? If not, the speed falls off since the torque is being limited by way of the supply current. That's the easiest of the lot to solve
As for some of the other material posted here, it is misleading at best. For example:
What is missing from that is the role of voltage in "forcing" the current through the motor. ...snip...
It is missing from the discussion because it does not exist. The voltage applied has zero effect on the available torque. A motor is not a resistor, amps are in no way proportional to applied volts.
If you have a specific motor and if you reduce the voltage from 6v to 3v then you CANNOT get the same current in the motor, never mind getting double the current.
Of course you can. Your point is only valid when discussing stall currents. A one amp motor running at 6V and 200ma is certainly capably of running at 3V and 400ma. It's a question of load behavior, not applied voltage.