That's not right!
An audio signal swings positive and negative. The Arduino can be damaged by the negative-half of the audio signal and/or the audio can be "damaged" (distorted).
At a minimum you should put a 10K "current limiting" resistor in series with the analog input.
The ATmega chip has built-in (low-power) "protection diodes" to protect the chip from negative voltages or voltages higher than VCC. Without the resistor the negative-half of the audio waveform (above about -0.5V) will be "shorted-out". The audio will be distorted and there's a chance of frying the protection diode and your Arduino.
Here is a more-complete circuit:
The 100K resistor keeps the analog input from "floating-up" when there is no connection, or if your audio circuit doesn't have a DC current path to ground. This resistor along with the 10K resistor makes a voltage divider, reducing the signal. To minimize that effect it should be at least 10 times the value of the 10K resistor.
The capacitor blocks any DC from your audio circuit (maybe not needed).
The Schottky diode is a better protection diode. It can handle a LOT more current, and it has a lower forward-voltage so the internal diode NEVER sees ANY current (also maybe not needed).
It's also possible to bias the input at 2.5V. This is actually a more common solution. That allows the audio to swing between +2.5V and -2.5V, and when it goes-into the Arduino it's going from 0-5V. (Then, you normally subtract-out the bias in software and if you don't want to change your software you don't want to bias the input.)