So now I'm supplying 5V to the 3.3V pin of the Velleman module and everything works perfectly.
This makes no sense.The datasheet you linked shows no 3.3V just 5V pin.
I know, but that's the official manual of the module. You can see the 3,3V pin on the image, and it is mentioned in the specifications below the table. I didn't expect it to solve my problem, but while I was looking up other SD Card modules, one of the comments was that the 3,3V and 5V pin labeling was wrong. So I figured why not try it as well with this one.