I ran into some more odd problems, like my code shrinking down to half it's size, when changing an = to a +=. The program also stopped functioning after this.
This was genuinly weird, so I tried the code on a Uno, same problem, so something really weird had to be going on.
The problem ended up being the extra while loop I added (to save those 10 precious clockcycles), after removing this, the code always compiled to the correct (large) amount of bytes.
I still don't really get what was going on, except for that the compiler optimized out all my code, but at least know where it came from.