Yes, the key insight you need is that (inertial) torque is proportional to acceleration - the speed isn't relevant, only the acceleration rate, and without speed ramping the acceleration you are trying to get is notionally infinite, so the torque needed is way too large to be possible.

torque = moment of inertia x angular acceleration (in Nm, kg m^2 and rad/s/s respectively)

The torque of a motor goes with 3rd power of size, the moment of inertia with 5th power of size, so the larger the motor the less angular acceleration it can provide.