Okay, I did some debugging and concluded that when I read the left or right channel of the audio signal through A0 input of arduino, I read "0" all the time. However, when the end of the cable connected to the A0 input is floating, there is some random noise and when I touch the cable it changes prominently.
The thing is, I get the audio signal from laptop's audio output and channel it to the speaker over breadboard, meaning the signal is there on the breadboard and going to the speaker without any problem, but when it comes to reading it through an Analog input, there is nothing.