A few cheap meter combines continuity and diode test, usually with a fairly bad result.
My question would be if that might be a sign for some error in my soldering or if the capacitors are just leaking current.
More than "a few" actually! Most that do not auto-range.
The capacitors will not be leaking current. If you put your fingers across them, that will leak current. And of course each time you swap the polarity applied to the capacitor it will draw current as it discharges and charges again.