but when I connected the audio to the Arduino input I noticed that my temperature readings went nuts.
Make sure that the audio input is not driving the analogue input pin outside the supply rails, especially make sure it is not taking the input negative. This can happen with an AC coupled signal. If this is the case then you can use some catcher diodes.
There is a bit of cross talk caused by the input multiplexer on the A/D but this is normally only apparent with high impedance inputs.
noticed that a high light value also influenced the temperature values
Make sure that your analogue reference is stable and you are not pulling it in any way. You could also try adding a capacitor to the Aref pin on the arduino.
Also make sure of the good stuff like common grounds.