where the sensor is a piece of metal behind some cardboard and my arduino is powered by my laptop. The connection to the laptop is not ideal as it affects the capacitance.
The minimal code to reproduce my problem:
There is a piece of metal where I am holding my hand. Things are hold in place with scotch-tape to prevent them from moving.
As far as I understand the library measures the capacitance of the circuit (see the documentation) and this capacitance is affected by any capacitance connected to the circuit. Here connected doesn't have to be an inductive connection but it can also be just a capacitive connection (I hope that makes sense). So the capacitance is different if I run it on a battery or if I connect it to my laptop. You can even see in the serial plotter whether your hands are on the keyboard of the laptop or not.
Also consider this advice from the library documentation:
"Another solution that seems to have worked well on at least one installation, is to run a foil ground plane under the sensor foil (insulated by plastic, paper, etc.), and connected by a wire to ground. This worked really well to stabilize sensor values and also seemed to dramatically increase sensor sensitivity. "
Also I think the sensor wire is much too long. That's just a feeling, I can't elaborate much without trying it myself. It's based on viewing it as an antenna, as suggested in reply #6.
The library is not designed to provide linear long term readings. It's set up to detect hand movements.