Ethernet2 (UDP) SPI transfers have a lot of dead time

Seems:
SPI.begin();
is required, shoved it in setup and it now works. I guess the Ethernet libraries are doing this and not
SPI.end();
when they are finished, making my other code work.