You haven't really clarified what you mean by the delay "when there's nothing on the other end". If the remote device is powered down then I suspect the problem is related to ARP.
Although at layer 3 (IP) you send to an IP address, at layer 2 (ethernet) your packet needs to have the MAC address of the destination in it in order to be sent. With your destination device on the local subnet, your source device will try to get the receiver's MAC address through ARP resolution. If it's not there, the ARP will fail and the packet won't be sent. The ARP request is likely to have a timeout associated with it that may be the cause of the delay.