I believe it is risky, it [u]may[/u] be OK. However is it not a recommended condition.
There are two considerations:
1) What is the current required by your SSR input if it were connected to 5V? Does it exceed the recommended for the due. If yes you can stop here and consider the Mosfet
2) If the above is "OK" then we have to look at the input circuit of the SSR. It likely is a diode and a resistor. If the input draws little or no current when connected to 5V - 3.3v = 1.7 Volts then you might get away with it. I'm guessing on what "little" current is but I would guess less than 200 to 500 µA.
I realize that the answers to the above questions require some testing, I don't know a way around it.
Perhaps if you has a cheap china ProMini running a 3.3 volts you could perform a test with it. If you damage the ProMini you only loose a few $$