It's OK to connect the -Vcc pin to ground but you will get the problems you are observing. That is because "regular" op-amps (like the TL081) cannot drive outputs all the way up to +Vcc or all the way down to -Vcc. That is, they do not have "rail-to-rail outputs" (+Vcc and -Vcc are called the supply "rails").
An op-amp like the TLV2371 will work better in this application as it has both rail-to-rail outputs and rail-to-rail inputs (a.k.a. RRIO).
The connections you describe should work if you use an RRIO op-amp.
--
BeatVox: inexpensive sound shield for Arduino, 3.5mm stereo output, 512Kbyte FLASH