ESP32 loses flash contents when 5v PSU gets turned off

Other Hardware 3rd Party Boards
1

As in the topic. I put a sketch on a bare-bones ESP32 which is on a custom PCB.
Powering it through USB and removing power and connecting power again, everything is fine and the sketch keeps starting up as normal, no issue however many times i do this, however many times i re-upload the sketch, no issue.

When i connect it to a 5v PSU that is also powering the LED board that is being controlled by the ESP32 custom PCB it all starts up OK, everything works as expected, however on several occasions, when i remove mains power from the PSU and the whole thing shuts down, the moment that i reconnect mains power, the ESP32 turns out (appears) to be without code in flash, thoroughly empty.

Clearly something happens during the time that the PSU is shutting down. Some of the boards appear to be more sensitive than others and the issue appears to happen more often to an ESP32 that has had the firmware flashed recently (it appears that way at least)

Now of course the power supply setup is the obvious thing to look at, particularly since it is happening mainly during the PSU shutting down. Google (ai) is telling something similar, brown-out can cause this, but to it had not really happened until recently.

I am dropping the 5v down to 3.3v using an AMS1117 3.3v regulator which has ample heatsink (the PCB gets a bit warmish but nothing that would indicate that the setup is not up to the task) An input capacitor, and an output capacitor. Anyway, during operation where 5v is removed instantly, all is fine and there are no issues.

What i have tried so far and to no avail.

  • added extra capacitors to the 3.3v rail as close to the ESP32 as possible, 1x 1uF 1x 22uF
  • added a 1N4148 as a fly-back diode near the power input (thinking maybe reverse voltage is an issue)
  • added extra input capacitors (even going as far as adding a 1000uF to the 5v Input)
  • added a physical 10K pullup to GPIO 0 (flash mode strapping pin)
  • added 100nF capacitor to the enable pin (Which is pulled up with a 10K resistor)

And now i am more or less out of ideas.

Of course in some ways the (switching) PSU is the cause, but it should be possible to switch it off by removing mains power, without it causing the loss of flash contents and the 3.3v rail should remain fairly stable anyway and what else can be done other than adding capacitors ?

The ESP32(S) are bought from Ali Express, but they were built by Espressif to begin with and they did not leave their factory without passing QC as far as i know. (they do not sell their reject batches) I considered getting them as part of the PCBA but somehow that does involve buying a whole reel of 650 pieces, which ends up being a E2000,- investment.
They may be old, they may be left overs from other projects where people did buy a whole real but didn't use all units, it is of course hard to say, but that will be the case with any of the other sources one may find these kinds of units. It is most likely a factor, but somehow this is not something i can change or fiddle with right now.

So.. Any ideas ?

2

Interesting problem. How do you know for sure the sketch is gone?
Think about it as a start up problem using the PSU not a shutdown problem.
Think about PCB board cracks, soldering issues, faulty components or a too big capacitor.
As a former troubleshooter these are what immediately come to my mind

3

Can you post an annotated schematic of the circuit and links to technical information on any non-generic parts? Please include a link to the exact ESP32 module you are using, as there are many versions.

This may help:

Boot mode pin issues. The ESP32 boot depends on GPIO strapping pins: GPIO0, GPIO2, GPIO12, GPIO15.

If these pins float or are incorrect, the device may enter download mode or fail to boot, making it look like "empty flash".

Check:
Ensure proper pull-ups/downs as follows:

  • GPIO0 -> pulled HIGH for normal boot
  • GPIO2 -> pulled LOW or left floating (depends on module)
  • GPIO12 -> must be LOW for 3.3 V flash

Check the flash memory voltage. I believe it is expecting 1.8 V flash in some configurations. You may have a flash voltage mismatch (GPIO12 problem). If GPIO12 is HIGH at boot, the ESP32 switches flash to 1.8 V mode. Most boards use 3.3 V flash. This will result in unreadable flash that appears erased.

You must be sure the power is solid, as that is the primary failure cause. Note that the ESP32 draws short current spikes (300-500 mA) on startup and during Wi-Fi activity. If the voltage dips, flash reads can fail and the brownout detector may reset the chip mid-write, which can cause data corruption.

Use a stable 3.3 V regulator, not power from a USB TTL adapter. If not already present, add 10 uF and 0.1 uF decoupling capacitors close to the chip. Watch for brownout messages on the serial console:
Brownout detector was triggered.

4

I think some ESP32 varieties have built-in flash memory, but most use an external serial flash chip. If yours is the latter type, then it's possible that chip is the problem.

Here's a thread that seems unrelated because it's a different problem and is about the ESP8266, but I wonder if something similar may be going on with your ESP32s, particularly if they are from Aliexpress.

https://forum.arduino.cc/t/wemos-d1-mini-pro-deepsleep-fail/1394898

The solution in that case was to replace the flash chips with Winbonds from Digikey, but I think there is also a fix that consists of adding a pullup resistor somewhere.

Well, this may not be relevant, but I think it is exceedingly unlikely that flash contents could be erased by a power glitch.

5

The debug message that is thrown out repeated, constantly mentions invalid (pff can't quite remember and don't want to force another erase right now, doing a temp stress test making use of the glorious weather here) starting address or something, but basically it can't find the code to execute. (if you really need to know i can do it later)
Re-uploading the sketch fixes it.

No ! The problem only occurs after the PSU has shut down from the mains. If i upload through the UART under USB power, i can start up as many times as i want, and the first startup with the PSU is always OK.

I re-melted using a heat-gun all of the suspect solder joints. Those are mainly the GND connections since those connect to a copper pour heatsink, regardless, if i just remove and reconnect 5v to the board with the PSU running, there is no issue. I suspect the 5v PSU drop below a threshold and then skips above it again momentarily when shutting down which could cause something like that.

6

Put a scope on the output and see if it is rising slowly. Also what power supply is it. I am assuming all solder joints are proper.

7

Screenshot (487)
Screenshot (487)1294×876 21.2 KB

This is the whole thing, assume only one of the 74HCT02 is actually connected.
The parts are not very exotic.

GPIO 2, 12 & 15 have internal pullups and downs but as far as i know do not influence boot mode. GPIO 0 can be left floating, but as stated i added a 10K pullup just in case. Though the output to the Serial monitor does not state that it is on the wrong boot mode, and once GPIO 0 is pulled LOW during a power cycle, it's expected message does show up.

But you may be onto something here. I have a yellow (status) LED connected to GPIO 12, pulling it LOW i guess, but the pulldown is active as it is.

Sometimes i do wonder if the original post contains to much information and therefore it is not properly read by all the people that are trying to help.

8

Well i can't get underneath the hood anyway to see either way. There was a report from google saying that a particular type of flash was more susceptible to this kind of issue.

9

That's sort of what I was trying to say about a capacitor problem, the scope sugestion from @gilshultz should show that.

10

I suspect it's dropping and then rising again before dropping completely, It's a 60A 5v switching cheap PSU, but at least it's mine and i have it right here. The board should be able to deal with crappy PSUs, it may malfunction maybe yes, but it should not erase it's flash contents. That would look like broken, although easily fixable, so i would need make sure that doesn't happen.

I am considering creating a soft-start addition. Something that detects low voltage, switches a mosfet and on powerup waits a second or so before switching the mosfet again. An ATtiny13a may do the trick.

I am quite happy that at least i manage to re-create the issue repeatedly, which means i can also confirm that it has been solved at some point if i do.

Yeah the issue is a bit that the 5v rail is connected straight to the PSU, so any added input capacitors will not make much difference (considering the PSU is powering a lot of WS2812b's which will suck it dry in no time) The 3.3v rail should have enough, but as stated before i added a few more. Still if i remove 5v directly (i remove a jumper that allows for 5v or 12v operation) and reinsert the jumper, there is no issue.

11

I will need to make sure it can actually deal with what is happening right now.

12

Can you post an annotated schematic of the circuit and links to technical information on non generic parts? Include a link to the ESP32, there are many versions.
This may help:
Boot mode pin issues, theESP32 boot depends on GPIO strapping pins: GPIO0, GPIO2, GPIO12, GPIO15

If these float or are wrong It may enter download mode or fail to boot making it look like like “empty flash”

Check:
Ensure proper pull-ups/downs as follows:

  • GPIO0 → pulled HIGH for normal boot
  • GPIO2 → pulled LOW or floating (depends on module)
  • GPIO12 → must be LOW for 3.3 V flash
    Check the flash memory memory voltage, I beleive it is expecting 1.8V flash.

You must be sure the power is good, that is the primary failure cause. Note The ESP32 draws short current spikes (300–500 mA) on startup and Wi-Fi. If the voltage dips: Flash reads can fail, Brownout detector may reset mid-write, this may cause data corruption.

Be sure to use a table 3.3 V regulator (not USB TTL adapter power) If not present Add 10 µF + 0.1 µF decoupling caps close to the chip,

  • Watch for brownout messages on serial:
    • Brownout detector was triggered
13

You are using a lot of GPIOs, It may be there but I don’t see clear enforcement of proper states on: GPIO0, GPIO2, GPIO12, and GPIO15

If any of these are driven externally (your logic, SD, Ethernet, etc.) at boot random boot mode or flash voltage mismatch. Start here: Hard-set strapping pins with resistors:

  • GPIO0 - pull-up (10k)
  • GPIO2 - safe state (often pull-down or leave floating per module)
  • GPIO12 - must be pulled LOW (critical)
  • GPIO15 - pull-down (typical)

I don’t see t ight local decoupling at the ESP32 module, I see that Bulk caps exist, but placement matters more than value. No obvious 0.1 µF + 1 µF right at VCC pins . Long traces from regulator to ESP32 (likely). SPI routing is long and shared, keep them short.

Please Confirm 74HC02 is not LS then Add Input resistors or protection if signals are external

Note: , the transistor + button arrangement is a bit unconventional, my concern is this could cause slow rise and or glitching reset.

14

Can you clarify what happens if you turn off the 5V supply, but then power up from USB instead of from the 5V again? Is it still messed up, or does it run ok without having to flash the sketch again?

I forget who suggested it, but I wonder if the 5V supply, or the 3.3V regulated derivative, is just rising too slowly.

15

Yes that was my thought as well initially, so i added 1uF really close to the ESP32 (as stated before)
The Vcc trace to the ESP32 is really wide.

With the exception of GPIO 0, i use all of them more or less.
GPIO 0 can be left floating but has an internal pullup, and i did add one (as stated before)

GPIO 15 is pulled up (it's the SD CS) That disables/enables log output ?
GPIO 12 is pulled down through the LED
GPIO 2 is pulled down through a LED.

Only GPIO actually influences the boot mode on an ESP32 as far as i know.

[quote="gilshultz, post:13, topic:1442269"]
Long traces from regulator to ESP32 (likely)
[/quote] Not that long, but i had to compromise to provide heatsink to the ESP32 which is on the GND plain, where as the AMS1117 does on Vcc, so some physical separation is required. I anyway can not easily change things on the current setup, A different type of regulator could be an option but the distance from the regulators heatsink (Vcc) to the ESP is less than 2cm and is 1mm wide. I think i am within margin but i admit i forgot about the de-coupler and added one just in case. I somehow assumed that the ESP32 has one on there (and i actually still think it does, but you know what they say about assume)

As short as possible, and they are not shared, a bus each for the W5500 and the SD-card. There is now real way to make them any shorter. The W5500 is right next to the ESP32, the SD-card is a bit further away, it needed to be near an edge of the board as did the RJ45, and i prioritized the signal to the 74HCT02 and the output. The whole thing is quite snug at 50x85mm and there are parts on both sides of the PCB

There are of course things that could cause some issues, but those should and/or would show up during normal operation.

I do wish i would have remelted some of the solder joints before i cleaned up any excess flux, because that has become a little tricky now (live and learn) and the Vcc pin of the ESP32 in particular is hard to get to with C2 obstructing it's physical access quite a bit.

16

Once the unit does no longer power up properly, it doesn't power up properly at all anymore.
When switching of the PSU and turning it on again it does not always loses it's flash contents, but once it does, it's gone.

17

That is a clue. This is a very strange problem indeed. I still have no idea how the flash is erased. Maybe start there, what conditions cause that then back up from there to how that connects to a power event.

18

I thought i started out with that.

For 12v testing situations i actually have a switch on the 12v rail since the old PSU i had was also throwing out High voltage spikes and causing some issues if i would not step down to 3.3v through 5v (i had some older designs that were not quite ready for that)

But yes i am seeing a possible solution in adding a P-channel mosfet and measuring voltage on the 5v rail.

Not sure if i have anything lying around i'll have a look later.

19

But you haven't read out the entire contents of the flash memory after this happens? So it could be just one byte that's messed up, not a complete erasure?

The datasheet for the ESP32-WROOM-32 shown on your schematic says it does use a separate 4MB serial flash chip, which I assume is under the metal shield, so I guess there's nothing you can do about that. But this is really strange, particularly since you can flash the sketch again and it works fine. So it would seem there's nothing actually wrong with the flash. I'm just having trouble coming up with something happening during power down that could cause this. But if I understand correctly, it has to be during 5V power down, because if you power down, then boot from USB, it doesn't boot up, whereas you can boot up and power down repeatedly on USB with no problem.

20

Have you tried with a high quality stand alone PSU, or a Pi wall wart or an Apple high quality wall wart?