Start by dropping this completely unworkable idea. Just use an appropriate H-bridge to control the motor from the fixed supply. Pololu has the best selection of brushed DC motor drivers.
It would seem there is a problem with your circuit. Can you please draw and post a block diagram of exactly how all this wire up, including the switch you used to reverse the motor.
Should be obvious you cannot reverse the polarity of the input voltage to a buck converter and not expect smoke. Did you look at the spec sheet for the buck converter?
There is a motor driver in the rc car, the car runs fine. I want to limit the maximum RPM of the motor whether it be forward or reverse, assuming max voltage to the motor is 9v, I want to limit it to 6v, whether going forward or backwards. I'm pretty new at electronics
To do that properly, you would first need to reverse engineer whatever is in the car.
For any motor with variable load (as in a car going up and down slopes, for example) it is not possible to control the RPM by just limiting the voltage to a fixed value. You have to measure the actual RPM using an encoder and adjust the motor controller, usually the PWM value, to suit.
Why not simply replace the existing motor speed controller with a standard motor controller intended for Arduino-type applications? They are cheap, and you would have countless program examples to study and work with.
Why are you refusing to draw a diagram to show us how you have wired this up. Throw in a photograph or two so we can see that your wiring diagram matches what you have done.
We only kno what you tell us and so far you have told us precious little technical information.
RC motors typically require very high currents to operate - 10A to 50A sort of range, an LM2596 is limited to a few amps.
Secondly a DC-DC converter like the LM2596 is DC (direct current - ie one way only). Its not AC, so you cannot reverse the polarity on it without vaporizing all the semiconductors on it as you did.
You might have been able to use a DC-DC converter before the ESC, assuming a high-current rated DC-DC converter was used, but the ESC itself would have a minimum working voltage anyway so this would be very limited.
ESC stands for electronic speed control. So why are you not using it for that purpose in the first place?
A "simple potentiometer" at these current levels will fail instantly, probably issuing flames. Very high current potentiometer (such as a wirewound rheostat) might be usable, but its big, expensive, bulky, and wastes power.
Power control for motors is done via switch-mode power conversion, just like the ESC does.
