The biggest annoyance to me in the Arduino IDE is the lag in the tools menu. By default, every time the Tools menu is shown, the IDE re-scans the computer for serial ports. However, on my computer, I have a lot of virtual and bluetooth serial ports on my computer and this re-scan can take up to 2 minutes (during which the IDE is completely locked up). (This is especially annoying when I use my UNO and Decimeilia in tandem because it wastes nearly 10 minutes just switching boards and ports.)
I have hacked together a simple mod to the IDE so that I can disable the automatic scanning of ports and also trigger the re-scan manually. You can download and compile my source available here: https://github.com/oroszwj/Arduino