The math for this is trivially easy if you can do algebra.
First, the datasheet formula:
ADC is the independent variable since that's the result generated by the hardware (between 0 and 1023 inclusive). So now there's two unknowns left, VIN and VREF.
The normal way is to take a fixed VREF with VIN as the dependent variable that is solved for. This arrangement creates a direct linear relationship between the dependent and independent variables, so has less error when converted to a real unit.
But there is no reason that you can't use the formula the other way. A fixed VIN (the 1.1V internal bandgap) can be used to calculate a variable VREF (AVCC). The internal reference is convenient to use since it requires no extra hardware, but has poor tolerance (only about 10%). If greater accuracy is needed, a separate voltage reference chip can be used on one of the analog inputs to do the same job. The only thing you would need to do is adjust the formula.
A second option for measuring the power voltage is to use a fixed reference for VREF, and use a voltage divider to scale down the voltage to within the ADC's input range. This is necessary if you are measuring a different power rail than what is supplying AVCC, but is inconvenient for measuring the chip's own power supply since it needs extra hardware.