Does anyone know what I'm doing wrong, or why some boards seem to work great?
Clearly, making a major wiring mistake outside of what you think you are doing.
I think you appear to be saying that you are powering the 3.3 V Pro Mini from 12 V using its on-board regulator. That would be a bad idea as the regulator is not adequately heatsinked for 12 V operation, and the situation is even worse than for a 5 V version. Now whether that would in itself "blow" the regulator is a curious question, most regulators are supposed to enter thermal shutdown if overloaded, but - may not be adequately robust and of course, cheap Chinese (eBay) parts are questionable.
As Wawa points out, powering the input side of the relay board - "Vcc" - from 5 V is an excellent way to operate it with a 3.3 V Pro Mini without disabling the green LED as long as the 3.3 V supply is derived from that 5 V supply so it cannot be lost unless the 5 V supply is lost also. Arguably, using a (buck) regulated 5 V supply to feed Vcc on the relay board and "Vin" on the Pro Mini should be quite reliable (presuming the on-board regulator has a dropout voltage less then 1.7).
We still don't know which relay board you're using (asked in post#4).
If you have a board with 12volt coils, then it should NOT have a VCC jumper.
If it does, and you have the jumper in the wrong place, then the full 12volt is connected to VCC of the Nano.
He has explained that it does have that jumper - which is as you would expect since the 12 V version with the jumper works just fine, given that you have to use 12 V logic on the input. He also explains that he has removed the jumper, so there is no problem there.
So the 5v side (3.3v in my case) is only powering green led and led in opto-isolator. That wouldn't be enough current to blow the regulator, would it? How much can these regulators supply?
With the green LED disabled so you can (ugh!) feed it from 3.3 V, the opto having a 1.2 V drop and a 1k resistor ("102") - but check that! - each opto will draw 2.4 mA. That certainly will not damage the regulator. I'm punting for a wiring error. Or ...
Would I not have to use a 3.3v regulator on 3.3v pro mini?
For the Pro Mini itself, if you insist on running it on 3 V, yes. With the regulator removed, you can run it at 5 V for almost as much efficiency since the current draw is affected more by the lower clock speed than the operating voltage.
I use the 3.3v pro mini because it uses a lot less power than the 5v version.
The 3.3v relay modules look good, but I need ones with 12v coils.
Different relays will require much the same power to actuate, so it is the case that operating suitably specified relays from a linear regulator is much less efficient than directly from the primary supply voltage.
Is there an alternative regulator I could use with low power and high current?
Tricky. An adequately rated switchmode regulator dropping 12 to either 3.3 or 5 V as explained above should resolve the situation.
I'm still confused what's happening though considering the Arduino pins are only driving the led in opto-isolator.
Short of a recurrent wiring blunder, the problem will be overheating of the on-board regulator.