Can I run a 40A ESC off a switching mode power supply instead of using a LiPo?

I have a 40-Amp Pro Switch-Mode BEC Brushless ESC can I run this at 11.1V off a switching mode power supply like mine below?

That would depend on what amperage your motor pulls when it starts up; if greater than 20 amps (the rating of your power supply), it might trip the over-current protection on it. The ESC can supply up-to 40 amps; you have to have a power supply that can also provide what the motor needs (and probably a bit more, just to make up for losses).

If your motor is rated for around 14 volts, and you find that your power supply can't output what the motor needs in current - you might look into getting an automobile battery charger/starter (it must be capable of powering the starter of the automobile); most of these can supply 50-100 amps (for a short amount of time). Just make sure you put it into the "starter" mode first, and be very careful with the leads when the device is turned on...