Ok, the answer is kinda obvious, the IDE copies source files from the sketch directory to a temp folder, and compiles there. So any subfolders from the sketch folder do not exist in the temp folder.
Seems like the only way to do something similar is with libraries, but that is not a satisfactory solution. I think that the Arduino IDE is probably not a suitable tool for me.