If you really want things to stay in place you might want to try soldering!
The easy fix would be to use an electric glue gun. You can probably find one at a hardware store, they're usually really cheap. The good thing about that kind of glue is that it builds up, and it's electrically insulating.
i wouldn't recommend hot glue for that. you might want to change the wiring later... if the glue gets into the sockets it also might isolate the socket from your wire.
the sockets used on arduino are standardized. you can get primitive "plug-rows" from a local electronics dealer.
something like that:
they stick quite firmly in the sockets. just solder your wires to the end.