The shield schematic clearly says D10-13 are used for the SD card and A4-A5 are used for real time clock. Then Leonardo uses D0 and D1 for hardware serial. Use any other pins, including A0-A3 analog pins. D4 is NOT used by the shield. Nick said with a "usually" which means he did NOT read the schematic, AGAIN!!! 
Here it is:
You can also buy a backpack from my store to save all your pins:
http://www.inmojo.com/store/liudr-arduino-and-physics-gadgets/item/serial-lcd-back-pack---phi-panel/
Will do you a lot good compared with its price.
BTW, is your RTC working? I suppose not but let me know. I have the same shield but don't have Leonardo.