IOT Cloud configured Wifi Credentials not used by device

Hi developers,
In fault finding an intermittent connection/data issue to Arduino IOT cloud library from a generic ESP8266 programmed by Cloud App, I have found a concerning issue with WIFI connection changes. I changed the WIFI Connection in the Cloud Thing Setup page and uploaded by USB. The device still connected to the previous WIFI, but reported connecting to the new network. I checked the Sketch sketch secrets page and thingproperties.h which showed no issues.
To test the IOT cloud data connection/data issue, I have been running the device through a laptop hotspot WIFI with Wireshark recording the Hotspot network, and Putty recording the device serial. This was providing me with some connection information, but seemed to be different from normal access point connection, so I changed the WIFI SSID and secret in the Cloud Thing Setup page and uploaded by USB to a testing WIFI Access point that I can sniff. I know it loaded new code as the serial monitor said that the device was Connected to "new WIFI", but the Wireshark of the old WIFI hotspot clearly showed connection and packet flow each time the device is rebooted/power cycled.
I could ping the device on the old WIFI for about 10 seconds after the Connection is reported on Serial to the new WIFI, but then couldn't after that. The device does not connect to the Cloud IOT.
A second (without changing setup or code) compile and upload of the thing fixed the issue such that the device connected to the correct WIFI. Agent version is 1.7.0-87f097b.
This is very disconcerting as changing a device's WIFI from test to deployment environment may not work, but is reported as successful.

All I know is that I have always used the cloud editor and uploaded (I think it's OTA) and never had any cloud problems. I have noticed that people who complain about cloud development are almost always using both cloud and IDE. I know the tools are there to allow download/upload, but from the beginning, I was suspicious how well that might work.
My advice is to stay in the cloud and set the creds to what you want. Also, please note that there could be significant delays (as I once encountered) of at least 5 minutes in getting changes propagated to the board.

Hi @kbe1. The ESP8266 stores Wi-Fi settings in non-volatile memory. The board will automatically attempt to connect to the access point using the stored settings:

https://arduino-esp8266.readthedocs.io/en/2.5.0/esp8266wifi/generic-class.html#persistent

The "Arduino_ConnectionHandler" library, which is used to handle the network connections for Arduino Cloud Thing sketches checks to see if the board is already connected to the network, and only initializes a new connection if it doesn't already have one:

So if the board is able to connect autonomously to the Wi-Fi AP using its stored settings before the Thing program executes ArduinoCloud.begin(ArduinoIoTPreferredConnection); then you get the behavior you noticed.

Something that might be confusing is that the library always uses the SSID from SECRET_SSID in the "Connected to ..." message it prints to Serial Monitor, not the actual SSID the board is connected to. As I explained above, these will not necessarily be the same thing and in this case the message printed to Serial Monitor will be false.

This behavior is not specific to Arduino Cloud Thing sketches. Here is a minimal demonstration sketch that will produce the same behavior (except for printing the true SSID) if your board already has usable settings stored:

#include <ESP8266WiFi.h>

void setup() {
  Serial.begin(9600);
  Serial.println("Giving time for board to connect autonomously:");
  for (int countdown = 10; countdown >= 0; countdown--) {
    Serial.println(countdown);
    delay(1000);
  }
  if (WiFi.status() != WL_CONNECTED) {
    WiFi.begin("foo", "bar");  // These invalid settings will not be used if the board was able to connect using its stored settings.
    Serial.print("Connecting");
    while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
    }
    Serial.println();
  } else {
    Serial.println("Using existing connection.");
  }

  Serial.print("Connected to ");
  Serial.println(WiFi.SSID());
}

void loop() {}

If you would like to erase the stored Wi-Fi settings, upload this sketch to your ESP8266 board:

void setup() {
  Serial.begin(9600);
  Serial.println("Erasing stored Wi-Fi settings.");
  if(ESP.eraseConfig()){
    Serial.println("Done!");
  } else {
    Serial.println("FAILED");
  }
}

void loop() {}

If you would like to prevent the board from storing the settings again, add these lines to the top of the setup function in your Thing sketches (as well as any other non-Thing sketches where you call WiFi.begin):

WiFi.mode(WIFI_STA);
WiFi.persistent(false);