My way of converting binary to BCD usually is calculate its decimal value and then convert to BCD. For this formula, I cannot understand, anyone can explain it? Thanks!

there is not such this as a 'binary value' but rather Binary representation

the decimal numbers 0 - 999 are represented in binary as 0b0000 0000 0000 - 0b0011 1110 011. they BOTH have the same value!

as to your question:

all that its doing if breaking down the decimal value into its individual digits to give you the corresponding BDC value!

dg[2] = (X/10^(2-2))%10^2 //returns the remainder when dividing number by 100 (10 raised by 2) AFTER number was first divided by 1 (10 raised by 0)
dg[1] = (X/10^(2-1))%10^1 //returns the remainder when dividing number by 10 (10 raised by 1) AFTER number was first divided by 10 (10 raised by 1)
dg[0] = (X/10^(2-0))%10^0 //returns the remainder when dividing number by 1 (10 raised by 0) AFTER number was first divided by 100 (10 raised by 2)
( m = MSD = array size = 2 )

a care point though, in arduino syntax the '^' represents XOR, so you will not be getting the intended output if you use the formula as is in the arduino IDE environment. something like this would however return the same result on arduino:

long i_pow(int val, int pow){
long ret = 1;
if(pow<0) return 0;
for(int i =0; i<pow; ++i){
ret *=val;
}
return ret;
}
void setup() {
int dg[3];
int X = 567;
Serial.begin(115200);
dg[2] = (X/i_pow(10,2))%10; //hundreds digit value
dg[1] = (X/i_pow(10,1))%10; //tens digit value
dg[0] = (X/i_pow(10,0))%10; //units digit value
Serial.println(dg[2]);
Serial.println(dg[1]);
Serial.println(dg[0]);
}
void loop() {
// put your main code here, to run repeatedly:
}

Not sure tbh; the math is correct, maybe its the syntax that need to be further corrected as previously mentioned...

I provided a 'simpler' code to do just that, y not use that instead!

the general formula for the code shared would be:

dg[n] = (X/(10^n))%10
where 'n' is the digit column (starting with zero for 'units' column), '^' means 'raised to the power of', and '%' is the modulo operator