Why, exactly, do you need the VirtualWire library? What functionality does it support?
I assume that it is to communicate with the RF module - is that correct?
If that is the case then...
Which RF module are you using?
Is VirtualWire (or similar approach) the only option?
Is there any possibility of using an RF module which doesn't require VirtualWire e.g. the cheap and common nRF24L01+?
I'm not saying you have to get rid of VirtualWire (I don't even know if it actually the cause of your problems) - I am just trying to highlight other possibilities.