Battery voltage dropping upon connection

I'd really rather not use something like a AAA...plus I need 5V. What about an A23? I found that while searching the Internet. Seems like they're really only used for small transmitters and things of that nature that don't pull much current, but with it being 12V could that work if I regulated it down?

Sorry if these are ignorant questions...again I'm pretty unfamiliar with this kind of thing. Thanks.