It must be a noise problem, most likely my other Arduino has better caps or an overall better design.
Replacing the voltage regulator probably helped because some regulators can cope with noise better than others and so does powering the Arduino with a 9V battery. Soldering a wire directly from the voltage regulator's output tab to the cartslot also seems to help if powered from a battery.
I haven't fully understood it yet and also don't have the equipment to do better testing.