How does the SAMD21 behave when it runs out of RAM?

Neither, It is not done in hardware and microcontroller's do not have operating system's. There are what is called RTOS(real time operating sytem's) for microcontroller's i.e. FreeRTOS that I am not real familiar with but I'm pretty sure none of them have this feature.

As J-M-L said you have to implement this within the code yourself.