I think it happens because the Arduino uses the feeding tension level comparing to the batery level (witch is connected to A0), so it will always indicate a 1023 (digital) signal (equals 5V) till the battery level drops below 4.3V (the feeding boosted level).
Your logic is correct - You'll read 1023 when the voltage is at, or above, the ADC reference.
But you need to "re-calibrate" your calculations because if your reference is 4.3V, then 1023 = 4.3V (not 5V). Then of course, you need to figure-in your 1:11 voltage divider.
And, if the reference voltage is from the same battery you're trying to measure you can't measure it because you reference drops when the battery voltage drops.
The optional 1.1V reference will remain solid & regulated as long as Vcc is "reasonable", so if you switch to the 1.1V reference, analogReference (INTERNAL), and re-do your calculations (where 1023 = 1.1V before figuring-in your voltage divider) everything should work.