I looked through the Adafruit library and can see that it really is flaky. I have no confidence in the validity of the lux calculation, as it currently stands.
First of all a "luxmeter" attempts to mimic the human eye response to sunlight illumination, in this case by comparing IR and visible photodiode outputs to approximate the spectral sensitivity curve.
(1) If the spectrum of the ambient illumination is not reasonably similar in shape to the spectrum of sunlight, the results of the lux calculation aren't very meaningful.
(2) The Adafruit library subtracts one photodiode output from another, and also divides one channel into another without checking for division by zero. That is probably where the NaNs are coming from, and is extremely ill advised.
The offensive line:
// See: https://github.com/adafruit/Adafruit_TSL2591_Library/issues/14
lux = (((float)ch0 - (float)ch1)) * (1.0F - ((float)ch1 / (float)ch0)) / cpl;
(3) There are alternative lux calculations in the code, currently commented out, that do not attempt this illegal operation. Furthermore, they include coefficients that the manufacturer recommends to improve the results. For example:
// Original lux calculation (for reference sake)
// lux1 = ( (float)ch0 - (TSL2591_LUX_COEFB * (float)ch1) ) / cpl;
// lux2 = ( ( TSL2591_LUX_COEFC * (float)ch0 ) - ( TSL2591_LUX_COEFD *
// (float)ch1 ) ) / cpl; lux = lux1 > lux2 ? lux1 : lux2;
If you want to use this device to measure low light level, I suggest to not use the Adafruit lux calculation. Instead, set the integration time so that neither register reaches a specified maximum (there is an interrupt specifically to detect that event, so you can determine how long that is), and have the Arduino just report the two register values as well as the gain and integration time.
You can process those data on your laptop later, using a better algorithm, or just use them for what they are: broad band intensities as measured by a visible and IR photodiode, with known response characteristics.
Maybe you can get suggestions from the manufacturer on better algorithms for processing the data.