ESP8266, PWM, MOSFETs - periodic freeze

Other Hardware 3rd Party Boards
1

I'm running a program (with Arduino framework) on ESP8266, which polls data over WiFi once an hour. The polled data is displayed on 2x OLED displays (via I2C), and then the (software) PWM is adjusted to drive RGB Leds on a matching color. In between the data polls, the ESP8266 sits executing a delay(1h).

Problem:
After executing from hours to days, at a random point in time, my prototype freezes. The displays are not updated, and the PWM (RGB color) remains unchanged. I have 6 prototypes in my house, so it's easy to see when one has stopped updating.

Remarks:
a) The circuit doesn't heat up, or otherwise signal any clear issue.
b) The RGB color remains still, which means the PWM remains in execution.
c) The other prototypes run ok, with the same program, which should eliminate possibility for faulty code. I've also diagnosed there are no memory leaks.
d) Once frozen, and I quickly power off and on the ESP8266, the OLED displays don't come back to life. I need to wait for some time, which I assume indicates there's current somewhere which needed to discharge first.
e) Upon first boot, I collect and store the WiFi credentials to EEPROM. It's more an exception, but twice I've had those wiped out upon a reboot after a freeze.

I've also probed the circuit with my oscilloscope and couldn't see spikes or other fluctuation in the voltage levels. But as this goes outside my expertise, I wouldn't exclude the possibility.

Circuit:

Circuit
Circuit1280×883 99.3 KB

AMS1117 converting 5V to 3.3V for the ESP8266. Before V(in) I've tried 10uF to 20uF tantalum caps. After V(out) I've tried 100uF (up to 1000uF) electrolytic caps. (Should probably try 1uF ceramic in parallel with the electrolytic?) Then, I've tested with a few different 5V supplies to ensure the issue wasn't on that end.

3x 2N7000 MOSFETs to drive 4x RGB Leds with the PWM. I started off without gate resistors, adding first 10ohm, and gradually up to 300ohm. I believe they slightly improved the stability, but this might be just a placebo.

Because my 2x OLED Displays share the same I2C address, I have 2x 2N7000 MOSFETs to control which one receives the I2C SCK signal. Similar to above, I started off without gate resistors gradually adding it up to 300ohm. (What I forgot to draw in the schematic is 10k pull-ups on both the OLEDs' SCK lines.)

Then, 10k pull-ups for RST, EN, and Gpio0, and 10k pull-down for Gpio15.

Remarks:
a) The MOSFETs add some capacitance on the I2C SCK line, but it should be on the safe side. While the SDA line doesn't have that capacitance, I guess it may add some offset.
b) My program is running in "debug" mode, writing out on the Serial RX/TX which are floating on my prototypes. Not sure if that may add some issues.

Other:
My first prototypes were without gate resistors on the MOSFETs. Interestingly, the one ESP8266 that has been executing flawlessly now for almost 2 months is from this batch.

I noticed Gpio0 can't produce 0V Low, but gives about 1.7V. Later, I connected the MOSFET gate to Gpio13 instead. With my first prototypes, a 10k pull-up on Gpio2 also seemed to destabilize the prototype (although, I'm not sure it was only that).

IMPORTANT:
The setup runs flawlessly from a development board to breadboard with jumper cables. The problems are only when I move to a protoboard with soldered and shorter (in length) connections. (The development board has also a slightly different set of components.)

Overall, more than fixing the problem, I'm interested in understanding it - as I'm doing this to learn electronics. But after banging my head to ChatGPT, books, and all other material for a month, I've come to conclude I needed someone to help me out.

I believe the answers lie in somewhere around these two observations:

  1. With a quick reboot, the OLED displays don't come to life.
  2. On a breadboard the setup runs smooth (and allows for that quick reboot).
2

Were the polarized capacitors and the voltage regulator moved from the breadboard, where they worked, to the protoboard. I don't mean the same part numbers, I mean the same parts. Just wondering if any of those might be bad or at least flaky. Otherwise it's hard to figure out if as you say the software is known to be identical (and I assume all the settings for flashing the code were the same).

3

No, not the exact same parts. But by creating 6 prototypes instead of 1, I was hoping to exclude the flaky parts from the picture. Nonetheless, it remains a possibility.

Then, as the faulty behavior on all the prototypes (except one) seems somewhat rational and deterministic, it makes me believe the problem wasn't in the circuit but in my understanding of electronics. What puzzles me the most is the "discharge time" needed upon the reboot, after a prototype had frozen.

I flash the ESP8266 chips with a development board, which I have wired up with the breadboard setup. On that one, the "discharge time" isn't needed, and I can toggle the ESP8266 as I wish. I've also ran the project on that one for more than a week, uninterrupted.

The development board is some cheap Chinese one, which has the same AMS1117 regulator on it (with a single, I assume tantalum, capacitor). It shouldn't be much a different setup from my prototypes, except for the longer jumper cables and their more loose connections.

One possible explanation I was thinking of, could be in the MOSFETs. Where I guess it was a possibility the ESP8266 turned 3x MOSFET Gates high the same time, at which point they could draw too much current, combined, for a nanosecond and destabilize the ESP8266. But this I thought would've been resolved by adding the gate resistors.

4

That is probably the best clue, somewhere/somehow the soldered board has some sort of issue like cold solder joint, whisker, some other high resistance short. Use a hot iron and 'touch up' every solder joint. Make sure the wires at each joint are secured with something while touching up. Use a solder cleaner and scrub the board clean.

5

Would a poor solder joint explain for the "discharge time" I've observed with the OLED displays? That puzzles me, and I'd be really happy to understanding the phenomena behind it.

But maybe I'd better start building my 7th prototype, just this time with the uttermost care on the solder joints. After banging my head to a wall for a month, I'm not sure whether to be happy or sad if that solved my problem. :slightly_smiling_face:

6

I find that a medium chisel tip is best. Many people never change the conical bit that came with the iron, but it is almost the worst choice. Look at it and imagine how much surface contact there is between the tip and the wire (not so bad) or the hdr pin/through hole. I use a hot setting of 700F/370C. For fine work I use 0.015"/0.38mm 2.2% flux Sn63 Pb37. It is No-Clean Water Washable. For larger work I use 0.8mm 1.5-2% flux Sn60 Pb40. I almost always use a liquid flux in small/medium plastic squeeze bottle with hollow needle applicator.

7

What does that mean?

Aside from what's been suggested, I'd recommend you investigate more with your scope, comparing what signals are present on a working circuit with what you see on a frozen circuit. And use your meter to compare voltages at various points in the circuit.

If one of the displays froze up and became unresponsive, how does your code react to that? You said the 8266 doesn't heat up. Is that true of the displays as well?

8

The development board is a 3rd party product with a snap-fit for the ESP, and a USB-to-Serial converter for flashing. The layout looks similar to the NodeMcuV2, while with less SMDs (to minimize manufacturing costs, I suppose). Then, my prototypes don't have the circuitry for flashing, but are designed to just run the ESP8266 in normal operation.

I don't know if the circuitry for flashing made a relevant difference, when running the ESP in normal mode on the board. But if something, the Serial TX and RX pins might not float - like they do on my prototypes.

Thinking of it, maybe I should remove the development board from the picture, by soldering jumper cables directly to a bare ESP8266 and hooking it up with the breadboard setup. It would be interesting if I managed to reproduce the issue there.

If one of the displays froze ..., how does your code react to that?

Good question, I will need to think of a test setup where I can monitor the serial for several days to capture that. Now, I've debugged them only after a reboot. Maybe I'll add some log collection over WiFi as well.

But from the RGB Leds I can see that the software PWM remains in execution. While as their duty cycles don't change anymore, it indicates the main program itself is not running as expected.

You said the 8266 doesn't heat up. Is that true of the displays as well?

The displays don't heat up either.

I'd recommend you investigate more with your scope

I will try this if I can't find anything from monitoring the code execution, starting from around the MOSFETs and the I2C lines.

Something I forgot to mention before, which might be relevant, is that the other display always lights up a few pixels in one of its corners. As it happens on all my prototypes, it probably indicates some issues with my I2C lines as well. (To add, my code always waits 300ms before and after I changed a MOSFET state.)

Question

Something I'm still very curious about, would be to try understand why the displays fail to start, when my prototype is powered off and back on without waiting minutes in between. Because from the RGB Leds, I can see my ESP is executing the program as expected.

9

Rather than looking at the Caps i would be interested to see what package you are using and how much heatsink the regulator has.

This is of course not very practical and in regards to the other observation with them not resetting properly, a possible cause.

No there is no problem with that.

All good, the one on GPIO 0 is technically speaking not required. The strapping pins GPIO 0, 1(TX) & 2 can be either pulled up or free-floating at boot.

Which does make me take note of the use of GPIO 2. It may be that it would be wiser to use a different pin for driving the mosfet, although you have reported that the ESP goes into it's normal mode, so i guess it's OK to use it as you do.

There is another thing relating to GPIO pins that may be of relevance. During the startup of the ESP some of the GPIO pins oscillate or go first HIGH or LOW and then the opposite way.

This may influence the OLED. Also it may help to show us which OLED you are using and well without the code there is not much we can say about how you have implemented both of them on the same address. I am not experienced with I2C so i can not be of much help in that direction, but having 2 devices with the same address is not a part of the design as far as i can tell. You could just remove one of them and see if the problem persists.

Some of the ESP barebones actually have some pullup & pulldown resistors on them like the ESP-07S hidden underneath the hood.

There may be something in the OLED that keeps alive on a very small amount of power and doesn't fully discharge to reset. This may be part of the I2C part of it who knows. I assume you completely cut the power both Vcc & GND.

My best guess is that the AMS1117 doesn't have enough heatsink for the current demand since that is the main thing that could be different from the devboard you used for development. Having 2 OLEDs with the same address obviously doesn't help and it would be nice to see the details on these and see if there is a better way of resetting them in some way to counteract the quick restart issues you have. Actually being able to cut power (or GND) to those could be helpful. They may even just be influencing each other when they start-up.

10

Then obviously there is a problem with that particular prototype.
Maybe a bad solder joint, bad part, wrong part, part in backwards.

11

The devboard has the AMS1117 regulator, which looks identical to those I have on my prototypes. It's a SOT-223 package. I haven't noticed the regulators catching up any heat at all, when I've "monitored" them with my finger.

Something I've thought of, is the current draw of the WiFi on the ESP. Whether that can momentarily exhaust the regulator, which then exhausts the ESP itself, causing it to halt the program execution (while without stopping the software PWM).

The OLEDs are cheap Chinese ones. Same as / similar to: 1.3" display and 0.91" display.

The shared I2C address is solved with the MOSFETs, which allow me to enable/disable the SCK line of a display. This way the displays never receive the I2C communications the same time, and their addresses become irrelevant in that matter.

The drawback is if the MOSFETs interfered with the I2C too much (i.e., now a different capacitance on SDA and SCK lines). But that doesn't seem to have been an issue. The I2C should also have a timeout, so such issues shouldn't be able to freeze my program entirely.

I don't know then, whether the MOSFETs can add something in the circuit which then prevents the displays from properly discharging (upon reboot, etc.).

What could prevent a display from discharging? Why doesn't it happen with the devboard setup? I would assume this can be the issue, but I don't know how to start solving it in my circuit.

Then, what causes the displays to "charge-up" in the first place? And is that something what can destabilize the ESP and freeze it from execution?

12

Apologies for my message being a bit unclear on that part.

What I meant to say is that I have 6 identical prototypes running in parallel. When one has stopped working, the others still continue normally. As they're all executing the same program, it shouldn't be then a programmatic error.

While if I ran them for a month, eventually none of them would work anymore. But there isn't any clear pattern to how or when that happens. Instead, the timing appears random.

For example, a prototype that had failed after 12 hours of execution, might after a reboot execute for a week without any issue. Or a prototype that had executed for 2 weeks, might next fail in just a few days.

Also, usually their displays fail to start-up on a boot, if I didn't wait for several minutes in between. This happens also if I rebooted an operational prototype, which hadn't yet frozen.

13

There's a problem right there. A 2N700 is a bad choice of MOSFET in most situations. RDS(on) spec'd at 4.5 V and up at 5.3 Ohm.. at top of that you use 3.3 V to drive them.

To address the other problem; What about multiple I2C buses instead, or are you short of I/O pins?

14

They make I2C multiplexers for exactly this problem. Take a look at the PCA9540B. It's an 8-pin chip, but it doesn't come in DIP, unfortunately.

If all of the circuits fail eventually at some random time, then perhaps it's something in the design that's operating very near the edge of crapping out, a line that it eventually crosses over under some stress. If a failed display really has to be shut down for minutes before it will restart, heat comes to mind as the most likely reason. Have you tried running one of these devices with a small fan on it all the time? Or do you have any cooling spray that might wake the display up early?

It just seems to me that if a display stops responding to I2C, the main processor may still be fine, but may simply be waiting forever for a response - depending on how your code and the library work. Have you tried suddenly disconnecting a display to see if the PWM freezes?

Also, you should put your scope on the output of the 1117 and see what it looks like during radio transmission. Put it in Single mode, with the falling edge trigger set just below 3.3V, and see if it ever triggers, but particularly during transmission.

15

The devboard has a heatsink using copper-fill which takes up a significant part of the PCB. Have you done the same ?

That tends to cause issues when you do not provide the AMS1117 with at least some heatsink other than the small tab.

While a voltage drop may not result in the ESP actually dropping out, it may result in the mosfets not working properly.

Well that is also how i see it. If you re-invent the wheel, but it causes issues, there is a good chance that the original wheel is the way to go.

My suggestion is to provide at least a few square cm of heatsink as a copper fill section on the PCB so the AMS1117 can provide something close to it's maximum 600mA. It is only dropping down from 5v so not much is required, but just the tab may cause dropouts when current draw exceeds 200mA
And use an I2S multiplexer. And then see if the problem persists.

16

Thinking about it, if I were going to use the mosfets, I think I would switch the data line, not the clock line. In I2C, I believe a slave sees drop on the data line while the clock line is high as a Start condition. It that's correct, then the non-selected display would see repeated Starts that don't go anywhere. But if data never goes low, then clock could be continuously fluttering, but there would be no effect on the display. Well, just grasping at straws.

Edit: And can you confirm there is at least one pullup resistor on the SDA line in your current circuit?

17

I never suggested that's the problem :upside_down_face:
However, OP uses two of these FET's to toggle I2C lines, which in theory could mess up I2C and the ESP.

Short story: Those 2n700 should be replaced to secure the operation.
Long story: All 2n700 should be replaced :slight_smile:

18

2N7000

19

Re switching only one I2c line, I knew I had looked at this before:

https://forum.arduino.cc/t/multiplexing-i2c-both-data-and-clock/1195688/16

And I also think bipolar NPNs (2N3904) would give more crisp transitions than mosfets. MCU ENable output through a 100K resistor to the base, emitter to the MCU SDA output, collector to the display SDA pin, with a pullup resistor added if the display doesn't already have one, which it probably does. The transistor in this setup produces a non-inverting open-collector output, but without having the mosfet's capacitance and threshold voltage to deal with.

Anyway, switch SDA, not SCL (if a multiplexer is not available).

20

the original wheel

Perhaps a NAND gate arrangement. These were really useful in the bad old days.