Question about gain on sensor

Hi,
I'm using one of the adafruit light sensors to measure light at night:

Adafruit TSL2591

So far it seems great. I've been modifying the code for purpose (logging, time/date) and changing the sensitivity via acquisition time and gain.

The issue is I don't know how gain works, beyond that it amplifies the signal (in this case I assume voltage) coming in to make it stronger to give a signal when there wouldn't be a measurable one otherwise.

This page talks about gain: Details of TSL2591

Is there any downside of huge gain other than it being too sensitive under normal light conditions? My concern is when they state "though the last value, MAX, has limited use in the real world given the extreme amount of gain applied" they're suggesting the signal isn't 'real' at that point? The values I get with that gain are well within what I'm looking for.

Thanks
Dean

I don't know how gain works, beyond that it amplifies the signal

And noise.

they're suggesting the signal isn't 'real' at that point

You be the judge. Put the sensor in total darkness and measure the noise level at several gain values, up to maximum.

As some experimentation will reveal, there is a tradeoff between gain, noise and measurement range.

jremington:
And noise.

Thanks, that's what I was worried about.

jremington:
You be the judge. Put the sensor in total darkness and measure the noise level at several gain values, up to maximum.

As some experimentation will reveal, there is a tradeoff between gain, noise and measurement range.

Total darkness is surprisingly hard to get, I'll try harder.

I'm aiming to try and get measurements as low as the sensor specs can do: 0.00018 Lux. On the second highest gain I've got down to 0.00053 for sustained periods. It's a matter of whether it'll do lower on the same gain, which I won't know until I get a bunch of 'nan' readings, which is what I'm trying to avoid. At the moment I've been using a median over a period of an hour, measurements every second, to try and remove the effects of noise.

Thanks for the help.

until I get a bunch of 'nan' readings

That would be due to a program error, like dividing by zero, or numerical overflow. That can be avoided by using good programming practices, and an understanding of the numerical limitations.

jremington:
That would be due to a program error, like dividing by zero, or numerical overflow. That can be avoided by using good programming practices, and an understanding of the numerical limitations.

I'm not sure what's going on then. It seems to massively increase when the gain is set to max, and never happens when the gain and acquisition time is normal for the light level. Nothing else is changed except the gain.

I've attached a screenshot of 2 lines of data. Left one is with high gain, right one is max gain. Nothing else about the code is different. This is in a box, in a closed drawer.

Post the code, using code tags.

The numbers in your screen shot are likely representative of the noise level, so without looking at the code, I'd guess 0.001 lux is the lowest light level you could expect to measure. The light of the full moon is typically 0.05 to 0.1 lux.

Looks like the code I'm using for logging etc is too long to fit in under the 9000 character limit. This is the measurement part of it though, straight from the adafruit site, with the max gain line added

/* TSL2591 Digital Light Sensor */
/* Dynamic Range: 600M:1 */
/* Maximum Lux: 88K */

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include "Adafruit_TSL2591.h"

// Example for demonstrating the TSL2591 library - public domain!

// connect SCL to I2C Clock
// connect SDA to I2C Data
// connect Vin to 3.3-5V DC
// connect GROUND to common ground

Adafruit_TSL2591 tsl = Adafruit_TSL2591(2591); // pass in a number for the sensor identifier (for your use later)

/**************************************************************************/
/*
    Displays some basic information on this sensor from the unified
    sensor API sensor_t type (see Adafruit_Sensor for more information)
*/
/**************************************************************************/
void displaySensorDetails(void)
{
  sensor_t sensor;
  tsl.getSensor(&sensor);
  Serial.println(F("------------------------------------"));
  Serial.print  (F("Sensor:       ")); Serial.println(sensor.name);
  Serial.print  (F("Driver Ver:   ")); Serial.println(sensor.version);
  Serial.print  (F("Unique ID:    ")); Serial.println(sensor.sensor_id);
  Serial.print  (F("Max Value:    ")); Serial.print(sensor.max_value); Serial.println(F(" lux"));
  Serial.print  (F("Min Value:    ")); Serial.print(sensor.min_value); Serial.println(F(" lux"));
  Serial.print  (F("Resolution:   ")); Serial.print(sensor.resolution, 4); Serial.println(F(" lux"));  
  Serial.println(F("------------------------------------"));
  Serial.println(F(""));
  delay(500);
}

/**************************************************************************/
/*
    Configures the gain and integration time for the TSL2591
*/
/**************************************************************************/
void configureSensor(void)
{
  // You can change the gain on the fly, to adapt to brighter/dimmer light situations
  //tsl.setGain(TSL2591_GAIN_LOW);    // 1x gain (bright light)
  //tsl.setGain(TSL2591_GAIN_MED);      // 25x gain
  //tsl.setGain(TSL2591_GAIN_HIGH);   // 428x gain
  tsl.setGain(TSL2591_GAIN_MAX); //9876x gain
  
  // Changing the integration time gives you a longer time over which to sense light
  // longer timelines are slower, but are good in very low light situtations!
  //tsl.setTiming(TSL2591_INTEGRATIONTIME_100MS);  // shortest integration time (bright light)
  // tsl.setTiming(TSL2591_INTEGRATIONTIME_200MS);
  //tsl.setTiming(TSL2591_INTEGRATIONTIME_300MS);
  // tsl.setTiming(TSL2591_INTEGRATIONTIME_400MS);
  // tsl.setTiming(TSL2591_INTEGRATIONTIME_500MS);
   tsl.setTiming(TSL2591_INTEGRATIONTIME_600MS);  // longest integration time (dim light)

  /* Display the gain and integration time for reference sake */  
  Serial.println(F("------------------------------------"));
  Serial.print  (F("Gain:         "));
  tsl2591Gain_t gain = tsl.getGain();
  switch(gain)
  {
    case TSL2591_GAIN_LOW:
      Serial.println(F("1x (Low)"));
      break;
    case TSL2591_GAIN_MED:
      Serial.println(F("25x (Medium)"));
      break;
    case TSL2591_GAIN_HIGH:
      Serial.println(F("428x (High)"));
      break;
    case TSL2591_GAIN_MAX:
      Serial.println(F("9876x (Max)"));
      break;
  }
  Serial.print  (F("Timing:       "));
  Serial.print((tsl.getTiming() + 1) * 100, DEC); 
  Serial.println(F(" ms"));
  Serial.println(F("------------------------------------"));
  Serial.println(F(""));
}


/**************************************************************************/
/*
    Program entry point for the Arduino sketch
*/
/**************************************************************************/
void setup(void) 
{
  Serial.begin(9600);
  
  Serial.println(F("Starting Adafruit TSL2591 Test!"));
  
  if (tsl.begin()) 
  {
    Serial.println(F("Found a TSL2591 sensor"));
  } 
  else 
  {
    Serial.println(F("No sensor found ... check your wiring?"));
    while (1);
  }
    
  /* Display some basic information on this sensor */
  displaySensorDetails();
  
  /* Configure the sensor */
  configureSensor();

  // Now we're ready to get readings ... move on to loop()!
}

/**************************************************************************/
/*
    Shows how to perform a basic read on visible, full spectrum or
    infrared light (returns raw 16-bit ADC values)
*/
/**************************************************************************/
void simpleRead(void)
{
  // Simple data read example. Just read the infrared, fullspecrtrum diode 
  // or 'visible' (difference between the two) channels.
  // This can take 100-600 milliseconds! Uncomment whichever of the following you want to read
  uint16_t x = tsl.getLuminosity(TSL2591_VISIBLE);
  //uint16_t x = tsl.getLuminosity(TSL2591_FULLSPECTRUM);
  //uint16_t x = tsl.getLuminosity(TSL2591_INFRARED);

  Serial.print(F("[ ")); Serial.print(millis()); Serial.print(F(" ms ] "));
  Serial.print(F("Luminosity: "));
  Serial.println(x, DEC);
}

/**************************************************************************/
/*
    Show how to read IR and Full Spectrum at once and convert to lux
*/
/**************************************************************************/
void advancedRead(void)
{
  // More advanced data read example. Read 32 bits with top 16 bits IR, bottom 16 bits full spectrum
  // That way you can do whatever math and comparisons you want!
  uint32_t lum = tsl.getFullLuminosity();
  uint16_t ir, full;
  ir = lum >> 16;
  full = lum & 0xFFFF;
  Serial.print(F("[ ")); Serial.print(millis()); Serial.print(F(" ms ] "));
  Serial.print(F("IR: ")); Serial.print(ir);  Serial.print(F("  "));
  Serial.print(F("Full: ")); Serial.print(full); Serial.print(F("  "));
  Serial.print(F("Visible: ")); Serial.print(full - ir); Serial.print(F("  "));
  Serial.print(F("Lux: ")); Serial.println(tsl.calculateLux(full, ir), 6);
}

/**************************************************************************/
/*
    Performs a read using the Adafruit Unified Sensor API.
*/
/**************************************************************************/
void unifiedSensorAPIRead(void)
{
  /* Get a new sensor event */ 
  sensors_event_t event;
  tsl.getEvent(&event);
 
  /* Display the results (light is measured in lux) */
  Serial.print(F("[ ")); Serial.print(event.timestamp); Serial.print(F(" ms ] "));
  if ((event.light == 0) |
      (event.light > 4294966000.0) | 
      (event.light <-4294966000.0))
  {
    /* If event.light = 0 lux the sensor is probably saturated */
    /* and no reliable data could be generated! */
    /* if event.light is +/- 4294967040 there was a float over/underflow */
    Serial.println(F("Invalid data (adjust gain or timing)"));
  }
  else
  {
    Serial.print(event.light); Serial.println(F(" lux"));
  }
}


/**************************************************************************/
/*
    Arduino loop function, called once 'setup' is complete (your own code
    should go here)
*/
/**************************************************************************/
void loop(void) 
{ 
  //simpleRead(); 
  advancedRead();
  // unifiedSensorAPIRead();
  
  delay(500);
}

Looks like the program author never tested this bit of code, which is claimed to check for over/underflow. The problem is in the library.

  if ((event.light == 0) |
      (event.light > 4294966000.0) |
      (event.light <-4294966000.0))
  {
    /* If event.light = 0 lux the sensor is probably saturated */
    /* and no reliable data could be generated! */
    /* if event.light is +/- 4294967040 there was a float over/underflow */

When it's over-saturated it reads '-1', but it tends to be pretty obvious, ie. when it's a high acquisition and gain and bright light.

The problem is in the library.

My apologies for my ignorance, but you're saying there's an issue where there's no way to distinguish whether a reading at low light is noise or a real reading?

The library evidently doesn't correctly handle floating point over/underflow.

You posted examples of noise in reply #4 -- the entries that are not "nan". Those seem to represent the photodiode dark current.

The numbers in your screen shot are likely representative of the noise level, so without looking at the code, I'd guess 0.001 lux is the lowest light level you could expect to measure. The light of the full moon is typically 0.05 to 0.1 lux.

Nazaar--:
At the moment I've been using a median over a period of an hour, measurements every second, to try and remove the effects of noise.

I've never heard of using the median to reduce noise in a repeated sampling setup - the mean is
used as it has well defined properties and is linear, so it won't distort or intermodulate actual signals
buried in the noise.

You posted examples of noise in reply #4 -- the entries that are not "nan". Those seem to represent the photodiode dark current.

Ok, so if the stated specs are that it can measure down to 0.00018, and the noise is about 0.0004, is it actually possible?

On a related note, how hard would it be to increase the acquisition time to, I assume, help reduce noise and allow lower readings?

I've never heard of using the median to reduce noise in a repeated sampling setup - the mean is
used as it has well defined properties and is linear, so it won't distort or intermodulate actual signals
buried in the noise.

I'm well aware of how out of my depth I am here, I'm a biologist trying to figure out how to do this, and am open to any suggestions from people who know more than me. I've been using the median because other light sensors I've used have 'blips' of obviously dodgy readings, so to filter them out I've taken the median previously.

Ok, so if the stated specs are that it can measure down to 0.00018, and the noise is about 0.0004, is it actually possible?

Not with the code supplied by Adafruit, it appears. If you ever see a NaN, it is not working correctly. I suggest to make a technical request to Adafruit.

On the other hand, you can do vastly better than this cheap sensor for very low light conditions, right down to counting individual photons. So why not try to define your project more carefully, before proceeding.

On the other hand, you can do vastly better than this cheap sensor for very low light conditions, right down to counting individual photons. So why not try to define your project more carefully, before proceeding.

I need to measure light at night. The two main purposes are distinguishing a clear lunar cycle, and measuring artificial light at night. From this thread it looks like these sensors will get most, if not all, of the lunar cycle in place with a fair bit of skyglow, but not in places without it.

The loggers need to be portable enough to be put underwater and not too expensive because this is ecology, therefore no money.

I've previously used the TSL237 sensors, but to get any kind of low light signal I needed to have an acquisition time of 5 minutes or so.

Open to any and all ideas at this point.

I've just posted on the adafruit forums, will see what they say.

Thanks for the help.

In the meantime would increasing the acquisition time help?

There are some comments on Github about the flakiness of the light intensity calculation with Adafruit's code. Clearly, it is not yet very well understood. The datasheet is very unclear and uninformative about the details of the sensor's internal operations.

More technical details are available at the manufacturer's site, but a quick glance through a few of the documents did not help.

If you are serious about your research, I suggest to NOT use this sensor at very low light levels, without carefully comparing the results to those obtained using a high quality commercial luxmeter designed for low light levels. Surely you can borrow one somewhere.

Thanks very much, I really appreciate the help. I'll read edits suggested.

If you are serious about your research, I suggest to NOT use this sensor at very low light levels, without carefully comparing the results to those obtained using a high quality commercial luxmeter designed for low light levels. Surely you can borrow one somewhere.

I'll have another look around, but the only one I've found previously is the Unihedron Sky Quality Meter, which uses the same sensor as I mentioned earlier. I'm sure the guy who made it knows a boatload more than I do, but I do know that his sensor isn't any better. I'll do some more intensive due diligence.

For what it's worth I edited the library to bump the acquisition time from 600ms to 900ms, and now when it's dark it either reads 0.0000, or nan, which is a vast improvement from saying it's not actually dark.

I looked through the Adafruit library and can see that it really is flaky. I have no confidence in the validity of the lux calculation, as it currently stands.

First of all a "luxmeter" attempts to mimic the human eye response to sunlight illumination, in this case by comparing IR and visible photodiode outputs to approximate the spectral sensitivity curve.

(1) If the spectrum of the ambient illumination is not reasonably similar in shape to the spectrum of sunlight, the results of the lux calculation aren't very meaningful.

(2) The Adafruit library subtracts one photodiode output from another, and also divides one channel into another without checking for division by zero. That is probably where the NaNs are coming from, and is extremely ill advised.

The offensive line:

  // See: https://github.com/adafruit/Adafruit_TSL2591_Library/issues/14
  lux = (((float)ch0 - (float)ch1)) * (1.0F - ((float)ch1 / (float)ch0)) / cpl;

(3) There are alternative lux calculations in the code, currently commented out, that do not attempt this illegal operation. Furthermore, they include coefficients that the manufacturer recommends to improve the results. For example:

  // Original lux calculation (for reference sake)
  // lux1 = ( (float)ch0 - (TSL2591_LUX_COEFB * (float)ch1) ) / cpl;
  // lux2 = ( ( TSL2591_LUX_COEFC * (float)ch0 ) - ( TSL2591_LUX_COEFD *
  // (float)ch1 ) ) / cpl; lux = lux1 > lux2 ? lux1 : lux2;

If you want to use this device to measure low light level, I suggest to not use the Adafruit lux calculation. Instead, set the integration time so that neither register reaches a specified maximum (there is an interrupt specifically to detect that event, so you can determine how long that is), and have the Arduino just report the two register values as well as the gain and integration time.

You can process those data on your laptop later, using a better algorithm, or just use them for what they are: broad band intensities as measured by a visible and IR photodiode, with known response characteristics.

Maybe you can get suggestions from the manufacturer on better algorithms for processing the data.

Ok, thanks, I'm learning a lot from this. I've never been game to look at a .h or .cpp before.

If you want to use this device to measure low light level, I suggest to not use the Adafruit lux calculation. Instead, set the integration time so that neither register reaches a specified maximum (there is an interrupt specifically to detect that event, so you can determine how long that is), and have the Arduino just report the two register values as well as the gain and integration time.

Ok, that makes sense, it already reads both, and I log both. I'm normally in the habit of keeping raw values for this reason; calculations change. I'd noticed that the raw numbers were really low already at the standard high acquisition time.

Re: the interrupt, do you mean this bit:

void Adafruit_TSL2591::registerInterrupt(
    uint16_t lowerThreshold, uint16_t upperThreshold,
    tsl2591Persist_t persist = TSL2591_PERSIST_ANY) {
  if (!_initialized) {
    if (!begin()) {
      return;
    }
  }

So bump up the acquisition time considerably when it's low light? It's already set to change across the day.

Maybe you can get suggestions from the manufacturer on better algorithms for processing the data.

Would you recommend approaching them directly, or leave it to the forums?

I would write directly to the manufacturer. Tell them you are a researcher, want to use the device in very low light levels, and ask for their recommendations.

So bump up the acquisition time considerably when it's low light?

That was not my point. You want to avoid having either of the two registers reach 65535, because the results won't be meaningful.

I would write directly to the manufacturer. Tell them you are a researcher, want to use the device in very low light levels, and ask for their recommendations.

I got a reply on the adafruit forums, their recommendation was to use this document: Custom lux formula which involves using an accurate lux meter to calibrate the formula. I can hire a lux meter by the looks of it, but finding one to do low light is going to be the struggle, as I'm guessing calibrating in normal light and applying the formula to low light isn't going to be too accurate. Still, probably more accurate than what is there.

That was not my point. You want to avoid having either of the two registers reach 65535, because the results won't be meaningful.

Ok, I'll avoid that. For reference at high gain and 600ms acquisition I'm getting counts of about 1 or 2 at 0.0005lux, so to get 65k I'd have to increase the acquisition time a lot. I was thinking up to 2000ms at most.