After weeks of Hologram tech support, and U-blox tech support the following solved the problem:
This is what did it! Response is now always under 3 seconds. Disables eDRX, the network can put the modem to sleep for a period of time after inactivity as I understand it.
AT+CEDRXS=0,4
This turns off power save mode (optional):
AT+CPSMS=0