PCB Prototyping - where to start?

Sounds like a lot for that board - but it also depends on how much vertical space you have to place those components. For starters you'll have to go for SMD components where possible, otherwise it really won't work. I really doubt you need four layers, there's a lot you can do in two. I've done what sounds like at least this complex a circuit on a two-layer PCB with only a handful of traces on the back.

The 13A MOSFETs will have to be TO220 packages, with heat sink. Even at very low RDS(ON) you dissipate about 4W per MOSFET. That's going to take half of your board, and you need sufficient vertical space AND ventilation (as it'll be inside your train: a small fan).

The other three MOSFETs can be SOT-23 so don't take much space. No issue there. The connectors for your smoke, lights, etc will take more space.

SD card reader also takes a lot of space on your PCB, unless you take one on a separate breakout board and place it vertical. Remember to allow for space somewhere to take your SD card out.

Then the Teensy. No experience really with that board but can't you somehow ditch the board and just use the processor? That also saves a lot of space.

You'll also probably have to go double sided (components on both sides) and go for the smallest packages possible (and yes you can hand solder 0.65mm pitch VSSOP packages, it's quite easy actually). As you put your Teensy on a socket you'll have a lot of free space under it. Great for all kinds of surface mount components.

For software, I'm using KiCAD. Does the job well.