Thanks guys! Got it working with help from you and guys at DIYAudio forums.
I also had to add (R5,R7) 47K resistors (towards ground) otherwise a little sound would bleed thru. And R8/R9 100K resistor to bleed charge out of C3/C4 so it doesn't kill my speaker.
Learned how to use LT Spice emulator in the process, very useful!
But by solving one issue I created another one. Now every time I change volume I hear loud pop (higher volume, louder the pop). I read that it's problem with generic digital pots

It looks like I might have to switch to PGA2310 or something similar designed for audio...