Okay, thank you Grumpy_Mike and tuxduino. I think I've got it.
I run under Windows 7.
When previously I tried to delete the folder whose name corresponds to the name of the sketch I was trying to get rid of, I got an error message, and when I exited that, another message saying something like "Windows Explorer is restarting" and Explorer then promptly proceeded to shut down by itself. I have now discovered that if I first delete the .ino file in the folder, the folder can then be deleted without a problem.
I am also able to rename now. The 'Save as (under new name)' and 'Delete (of sketch under old name)' procedure just seems a bit cumbersome though.