I tried different wire patterns but in like 30% of the cases it just doesn't work because the organs somehow do not properly touch the two contacts.
Yes that is not a surprise, it is difficult to get a reliable contact like that.
Options are
Use a magnetic organ and have a hall effect sensor underneath.
Use a photo reflective switch to detect the removal.