What about using one of those little keychain cameras like this one? Won't store the images on a removable memory card, but at least you can use USB to retrieve them.
You could also get a normal P&S and remove the LCD and case to get the size down. Of course the next bulkiest part is the optical train, and you can't remove that without sending image quality down the tubes. But there are some very slim P&S cameras out there for ~$100 now.