TCA9548A i2c multiplexer channel only holds on soft reset

Hello, I'm working on a device that uses a tca9548apwr multiplexer (MUX) to switch between 4 VL6180X LIDAR TOF sensors. I was having issues, so I went through troubleshooting, including i2c scanning sketches, and I have narrowed my issue down to a very specific condition. I can see the MUX no problem (0x70), and then when I change channels, nothing... UNLESS I soft-reset the ESP32 (pulling the EN pin low) after the channel was changed. Then, the channel actually engages and I can see the LIDAR through it (0x27). I have no problem using other i2c devices, as I am simultaneously using an i2c OLED screen on Wire1, I have tried switching it all around (putting my OLED on Wire and MUX on Wire1) etc. It always does the same thing.

Here is my scanner code. Nothing special, typical MUX code, standard settings for everything except for custom pin selection on the ESP32, but that does not seem to be causing any issues (it's an S3 so there are no longer any read-only pins).

// ESP32 I2C Scanner that also digs into every channel of a multiplexer
// currently set to channels 0-3 only instead of 0-7 because I left 4-7 floating and they take longer for nothing
// Based on code https://www.esp32.com/viewtopic.php?t=4742

#include <Wire.h>
#include <Adafruit_SSD1306.h>

byte count = 0, micount = 0, mccount = 0, snapshot = 0;
int channels = 4; //number of channels to scan, from 0 up to 7 (values 1 to 8; the scanner will stop after channel# channels - 1)

//OLED
#define SCL_1 10 //swap to 9 on V2 hardware version
#define SDA_1 9 //swap to 10 on V2 hardware version
#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 32 // OLED display height, in pixels
#define OLED_RESET     -1 // Reset pin # (or -1 if sharing Arduino reset pin)
//#define OLED_ADDRESS 0x3C

Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire1, OLED_RESET);

//TCA9548APWR Multiplexer for VL6180 lidars
// https://randomnerdtutorials.com/tca9548a-i2c-multiplexer-esp32-esp8266-arduino/
#define SCL_2 11
#define SDA_2 12
#define MULTIPLEXER_ADDRESS 0x70

void TCA9548A(uint8_t bus){
  Wire.beginTransmission(MULTIPLEXER_ADDRESS);  // TCA9548A address is 0x70
  Wire.write(1 << bus);          // send byte to select bus
  Wire.endTransmission();
}

void setup()
{
  Serial.begin(9600);
  Wire1.begin(SDA_1, SCL_1); //OLED
  Wire.begin(SDA_2, SCL_2); //Multiplexer

  //OLED setup
  if(!display.begin(SSD1306_SWITCHCAPVCC, 0x3C)) {
    Serial.println(F("SSD1306 allocation failed"));
    for(;;);
  }
  display.clearDisplay();
  display.setTextSize(1);
  display.setTextColor(WHITE);
  display.setCursor(0, 0);
  display.println(F("Hello, world!"));
  display.display();
  delay(1000);
}

void Scanner ()
{
  display.clearDisplay();
  display.setCursor(0, 0);
  display.println ("I2C scanner. Scanning ...");
  display.display();

  for (byte i = 8; i < 120; i++)
  {
    Wire.beginTransmission (i);          // Begin I2C transmission Address (i)
    if (Wire.endTransmission () == 0)  // Receive 0 = success (ACK response) 
    {
      display.clearDisplay();
      display.setCursor(0, 0);
      display.print ("Found address: ");
      display.print (i, DEC);
      display.print (" (0x");
      display.print (i, HEX);     // PCF8574 7 bit address
      display.println (")");
      display.display();
      delay(1000);
      count++;
      for (byte j = 0; j < 4; j++) //dig loop deeper for multiplexer
      {
      TCA9548A(j);
      delay(100);
      snapshot = micount;
        for (byte k = 8; k < 120; k++)
        {
          display.clearDisplay();
          display.setCursor(0, 0);
          display.print ("checking channel: ");
          display.print (j);
          display.display();
          
          Wire.beginTransmission (j);          // Begin I2C transmission Address (i)
          if (Wire.endTransmission () == 0)  // Receive 0 = success (ACK response) 
          {
            display.clearDisplay();
            display.setCursor(0, 0);
            display.print ("Found address: ");
            display.print (k, DEC);
            display.print (" (0x");
            display.print (k, HEX);     // PCF8574 7 bit address
            display.println (")");
            display.print ("on channel: ");
            display.print (j);
            display.display();
            delay(1000);
            micount++;
          }
        }
      if (snapshot < micount) {mccount++;}
      }
    }
  }
}
void loop()
{
  Scanner ();
  display.clearDisplay();
  display.setCursor(0, 0);
  display.print ("Found ");      
  display.print (count, DEC);        // numbers of devices
  display.println (" device(s).");
  display.print ("Found ");      
  display.print (micount, DEC);        // numbers of devices
  display.println (" sub-device(s).");
  display.display();
  display.print ("On ");      
  display.print (mccount, DEC);        // numbers of devices
  display.println (" channel(s).");
  display.display();
  while(1);
}

Why would my channel change not be taking unless I soft reset the ESP32? I have tried adding a long delay (1 second) after the channel selection, and it changes nothing.

This is how my multiplexer is setup

Thanks in advance!

Hi, I don't see the problem, but there is too much going on. I think that you can do easier and better tests and we need to know more.

Complexity that you have now: ESP32-S3 + OLED + two I2C buses + software + I2C mux + ToF sensors + wiring of the I2C bus.
What you need to test: I2C mux + ToF sensors.
Remove everything that could contribute to the problem. Yes, that is the ESP32-S3 as well. The ESP32 with its build environment is known to work, I don't know about the ESP32-S3.

Do you use modules or the bare chips ? Please give links to all the modules (a link to where you bought them). You could be using Adafruit modules for the OLED / ToF sensor / I2C multiplexer.

Can you show a photo with the wiring ?

How about the power ? Did you check all the voltages with a multimeter ?

Do you have a ESP32 ? The software for the ESP32-S3 could still have bugs. Can you buy a ESP32 ?

I think that a test is not fair when there is a OLED display connected, even if that is on the other I2C bus. So please remove the OLED and everything else from the ESP32-S3. Keep only the I2C multiplexer and the ToF sensors. Use the Serial Monitor for output. I even prefer that you use the default pins of the I2C bus (pin 21 for SDA, pin 22 for SCL for a ESP32, I don't know about the ESP32-S3).

I don't see the logic of your Scanner function. There are only four (sub) I2C buses to scan.
An extra test could be scanning the main I2C bus with no (sub) I2C bus, the I2C multiplexer has to disable all the (sub) I2C buses for that.

void TCA9548A_disable(uint8_t bus) {
  Wire.beginTransmission(MULTIPLEXER_ADDRESS);  // TCA9548A address is 0x70
  Wire.write(0x00);          // select no (sub) I2C bus
  Wire.endTransmission();
}

Please print the found I2C bus directly, because I don't understand the 'snapshot' and 'micount' variables.
You don't need a delay, everything should work immediately after switching the I2C multiplexer.

Code not tested:

Serial.println("Scanning main I2C bus only");
TCA9548A_disable();
for (int i=1; i<127; i++)
{
  Wire.beginTransmission(i);
  if (Wire.endTransmission () == 0)
  {
    Serial.print("0x");
    Serial.println(i, HEX);
  }
}

for (int j=0; j<4; j++)
{
  Serial.print("Scanning with (sub) I2C bus: ");
  Serial.println(j);
  TCA9548A(j);
  for (int i=1; i<127; i++)
  {
    Wire.beginTransmission(i);
    if (Wire.endTransmission () == 0)
    {
      Serial.print("0x");
      Serial.println(i, HEX);
    }
  }
}
Serial.println("Done");

Please show the full sketch, so I can see that there is no OLED code anymore. Don't even include the OLED library.

#include <Wire.h>

void setup()
{
  Serial.begin(115200);
  Serial.println("The sketch has started");
  Wire.begin();
}

Some platforms and some sensors do not work well with the I2C Scanner, but I have not read about problems of the VL6180X sensors with the I2C Scanner.

I'm using the bare chips, that's why I included my wiring diagram.

ToF chip is not the issue here, I haven't gotten that far yet. It's the channel switching on the MUX that is giving me issues. Doesn't matter what is behind the MUX, it's the fact that the channel change only takes hold after I soft reset the ESP32 which is puzzling to me. If I do a soft-reset after changing the MUX channel, the ToF sensor is perfectly visible and the MUX and ToF are working as intended... Until I try to change the channel again. So it's really the channel swapping on the TCA MUX that is my problem.

I am using the OLED because my USB implementation is giving me a hard time with the Serial readout (yet another thing I am troubleshooting, it's the first time I use USB-OTG).

I do have an ESP32-S2, but the MUX and ESP32-S3 chip are mounted to my PCB so I can't really swap it to my S2. Also makes changing the pins pretty difficult since everything is in PCB traces so it will have to wait to tomorrow (but I will try it).

Snapshot and micount are just counters that only go up if something is found on a sub-channel, it is purely passive data collection for totals because the OLED overwrites the addresses as it refreshes and I wanted to keep track of how many items it found if the screen was refreshing too fast for me to catch it all (yes, the serial monitor would be much better here!).

Anyways, I will be removing the chip from the PCB tomorrow and will make a temporary breakout board to test it with the ESP32-S2 which I know works with the serial scanner as well. Remove extra variables.

Do you have a schematic ?

Do you have 100nF decoupling capacitors ? The 10 µF doesn't do much for high frequency spikes.

Have you connected 3.3V to VCC of the TCA9548A by accident or did you read about VPASS ?

What is connected to MUXEN ? The RESET of the ESP32-S3 I assume ?
So a reset to the TCA9548A makes it accept I2C commands. That is interesting :thinking: Maybe it got somehow in a erroneous state. Pin 9, 10, 11 and 12 of the ESP32-S3 are common pins (not "strapping pins" or something else special). You could add a delay of 1 second in setup(). I'm thinking about a software compatibility Issue, because there is a long list of Issues: https://github.com/espressif/arduino-esp32/issues?q=is%3Aissue+%22esp32-s3%22

Thanks for all the info! Sure, I have a full schematic. Sorry for not posting it at first, I was thinking maybe it would just add to the confusion.

I will add the 100nF as you recommended. I should have some laying around from past projects. I usually have those on my MCU and power supplies, but I didn't think of putting it on the MUX, I was only putting a cap there to avoid brown outs, not to filter.

MUXEN is the reset pin of the MUX, and I just keep it high in this version of my design but I did change it so it's controllable by a GPIO in my next iteration. The ESP32-S3 en/reset pin is actually on a hardware button (full schematic below).

This probably won't help much because the wiring is kind of hard to follow when you can't zoom and look at the trace names, but just in case it does, here it is. Also I removed the planes for the 2.8V, GND, 3.3V, 5V, and 12V from the image so the traces are actually visible. You can assume that all the ratlines are actually connected properly to those power planes.

WOAH! Thanks for that link! I was not aware of those issues, I will do a bunch of reading today before I do anything else in my code or hardware for sure!

Just so that everything is available, here is the share-alike non commercial schematic and layout. https://oshwlab.com/rpb-team/laser-z-adjustor-v2 I figure if I'm getting community help, I should make the files available.

I don't know much about a PCB design, hopefully someone else will take a look.

I see signals side by side. Are those SCL and SDA ? The connector for the OLED has them also next to each other.
This is the official I2C standard: https://www.nxp.com/docs/en/user-guide/UM10204.pdf
Alternative link: https://www.pololu.com/file/0J435/UM10204.pdf
Go to page 54, read paragraph 7.5
As you can read, SDA and SCL happen to dislike each other. They can be near each other for 10cm, but they get into a hairy fight if it is longer.

The ESP needs also decoupling capacitors of 100nF and there should be a few more 100nF here and there. Do you know how noisy the DC-DC converters are ?

The MUXEN to a IO pin is weird. No one does that. No one needs that. It is weird and you have a problem with that mux. That means it is the first thing to look into. Can you keep the pullup resistor and cut the trace to IO45 ?

I see no other option than to build a test with a ESP32. The software for the ESP32 has been continuously been improved over the last years, it might even be more reliable than some official Arduino boards.
That is a "ESP32", not your ESP32-S2 :face_with_raised_eyebrow:

At the moment, MUXEN is not going to an IO (it's how the specification sheet recommends allowing for a quick reset in the event of a hang though). It's just pulled high at the moment.

Yes, I do need to add back the 100nF. Unfortunately I've made lots of fixes since I ordered this first test board and so since it's a live project it's not 100% reflective of what I actually have on hand. Lots is spiderlegged on my current board for example as I fixed things here and there.

The DC-DC are actually pretty great! But I'm still going to add some 100nF. Cheap insurance. I had them in the previous iteration, not sure why I removed them.... So good catch!

The longest i2c trace is 64mm for one of the LIDAR, but the MUX is only 35mm.

I'm pretty sure it's hard to find any non-S2 at this point. From what I understood (but this is somewhat limited knowledge I admit), the S2 replaced the original, and the S3 was an upgrade to the S2. Most of the time people don't specify because when you use a featherboard, it's pretty much identical, but when you use a WROOM (which as all the pins exposed) or the bare chip, the pins have changed a bit over time.

I'm thinking of just buying an adafruit MUX board to test my code in the first place. I've been at this for almost 2 weeks now and It's driving me mad.

Alright, decouplers and filters added back in and adafruit MUX ordered. Same chipset, so we will see how that goes! I'll report back tomorrow evening.

I see :open_book: Sorry, I was wrong. In case that one of the (sub) I2C buses is stuck, the device can be reset so the rest can continue.

The webshops still sell ESP32 boards, but when I look at mouser.com, then the original ESP32 is not there any more.

How about the VL6180X ?
This is just one of the many: https://community.st.com/s/question/0D53W000009g7b0SAA/vl6180x-stops-responding-on-i2c
Others mention the GPIO0 / XSHUTDOWN pin as well.

Each VL6180X needs a 100nF decoupling for the 2.8V.

You could lower the pullup resistors for SDA and SCL to each VL6180X.
At this moment it is 2.8V / 10k = 0.28mA sink current to pull a signal low.
I suggest 2k2 pullup resistors. It will make the whole project sturdier.

The main 3.3V I2C bus has 3.3V / 10k = 0.33mA
I suggest 2k7 there.

Does your OLED have internal pullup resistors ? You might need also extra pullup for SDA2 and SCL2.

Instead of using the second I2C bus for the OLED, you could connect the OLED to another (sub) I2C bus of the I2C multiplexer. It will not interfere with the other (sub) I2C buses. This is just an option, I don't know if it will by any good.

I hope you get more information by trying to change things in software and hardware. If the problem is consistent, then I don't know what to do.

The OLED is a pre-built module with the built-in pullups so that part should be fine in theory.

I'm not ignoring you, amazon decided to lose my adafruit MUX chip TWICE!!! lol I'll report back as soon as I can finally test that. I have a feeling it's my hardware configuration because i2c protocol is so simple and mostly fool proof (at least for just doing a port scan and making sure the components exist). Forcing a restart on the MUX VL6180X after the channel change is interesting though, especially since both spec sheets say those fix hangs, and they're instant / don't require additional delays.... could be a good "safety net" in my protocol.

The strong pullup on the VL6180X is a very good lead thought! Especially since the documentation has us leaving GPIO1 floating!!! page 10 https://www.st.com/resource/en/datasheet/vl6180x.pdf

Alright! Finally got my pre-built TCA9548A test board. I went with a ESP32 dev module (not S2 or S3) and used the default i2c pins (21 and 22).

I am still using an OLED for the output, reason being that I have trouble getting a reliable output on the Serial port. Sometimes it prints, sometimes it doesn't....

Anyways, I don't have any ready-to-plug i2c devices other than the OLED screens when it comes to testing. So I started with OLED + MUX on same port. It found both, no problem, but was not finding the OLED through the MUX (which was to be expected, same channel issue with the main OLED). All good so far.
I moved the main OLED to Wire1 (different pins). It works fine, TCA is still being found just fine on the &Wire pins and my OLED on &Wire1 still gives me the output. However, I am still not picking up the second OLED board I have plugged into SD2 and SC2 of the MUX. The soft reset "trick" also doesn't work now. Pressing EN when it's scanning the correct MUX channel does not allow it to reboot in the port 2 state.

I don't know what to say and no one else has joined this topic yet :sob:

The Serial port should work. If there is a problem with that, then you should fix that first.
The ESP32 has no usb-serial device, you can just use Serial.begin() and Serial.println().
The others with a usb device probably need while(!Serial);

If you don't connect anything to the ESP32 module, is the Serial output 100% reliable ? No one in the world has problems with that. It must work.

I suppose that we can say that 'Wire' and 'Wire1' are working reliable ?

A OLED display can be weird. A made a special note here that a OLED can disturb the I2C bus for others.
However, a OLED behind a I2C multiplexer should be no problem. It must work. I have no clue what could be wrong. The I2C multiplexer connects both I2C buses with mosfets, it is very simple. So I'm still thinking about a problem with the power or the GND or the RESET pin.

Yeah my issues with serial included with nothing plugged and no code except for my serial begin and serial println in startup and the a serial println in loop... So clearly I have to figure out where my problem is coming from. Even without reflashing, sometimes it prints sometimes not... But oled is always working. I'm wondering if it's a sync / baud speed issue.

Thanks for trying, I think I have to test every component (including cables and drivers) and make sure nothing is faulty and throwing up false problems. I guess I'll be busy for a while!

Thanks again for trying to help!

Okay now I feel a bit stupid! The second OLED was detected and works. The issue here was that I (erroneously) assumed that an i2c device would be detected automatically, but I actually had to initialize it before the scanner would successfully return that it had found it! So I will be changing my sub-channel scanner to run the full device initialization routine in the future...

This brings me back to testing my original device now that I have confirmed working code and confirmed good components and narrows down my issue to almost certainly being tied to my own MUX layout.

I ordered V2 hardware last night with all the fixes implemented. I should have a better starting point to keep troubleshooting. Thanks for all the input!

Okay. Hardware V2 is in. Everything flashes better than before, serial monitor now works as expected, but I am still not picking up my LIDAR sensors on the MUX. I really wish the MUX could send back a response to commands or something to acknowledge.

The Wire.endTransmission() return an error code and the Wire.requestFrom() returns the number of received bytes. With those two functions you can test if a I2C write or a I2C read was delivered to the Slave.

1 Like

Perfect, I'll have those printed to the serial monitor!

Thanks for the information. Sorry it took so long to get back to you, health stuff.

I'm getting
no matching function for call to 'TwoWire::requestFrom()'
when I try using the Wire commands (but Wire.begin() works just fine....)

I feel like I'm missing something obvious.

For some context, this is where I tried plugging it in.

TCA9548A_disable();
  TCA9548A(1);
  if (!sensorYN.begin()) {
    display.clearDisplay();
    display.setCursor(0, 0);
    display.print("Failed to boot sensorYN");
    Serial.println("Failed to boot sensorYN");
    Serial.println(Wire.requestFrom());
    display.display();
    delay(2000);
    }
    else{
    display.clearDisplay();
    display.setCursor(0, 0);
    display.print("Booted sensorYN");
    display.display();
    delay(2000); 
    }

If you have code that should work, then you can test the return values of those functions.
You can not just do a Wire.requestFrom() with no parameters, such function does not exist.

I have made an alternative explanation of the functions of the Wire library: Explanation of the functions of the Wire library · Koepel/How-to-use-the-Arduino-Wire-library Wiki · GitHub

1 Like