ESP32: How to get ESP to pair with bluetooth LE server?

[PORTB]

Hello,
I am making a project where I use an ESP32 to turn on/off a light that has bluetooth low energy controls. I am having problems trying to get the ESP to read/write characteristics of the light switch (the light switch acts as the server, the esp is the client).

I did actually manage to make it work with one board, but not on any other board or any other light switch. The problem is I don't know how I managed to do it. This also confirms that the UUIDs are correct.

I think the problem is that the light switch device doesn't allow unpaired devices to read/write characteristics, because using the nrf connect app on my android phone, notificaiton events work fine. Notifications work fine on the esp also. But when I try to read or write the characteristic, the app requires that I pair with the device. I suspect I need to do this with the ESP, but I don't know how.

I found some information about controlling the light switch here also: Read/write Clipsal smart switches · GitHub

my code, if it's needed

#include <Arduino.h>
#include <BLEDevice.h>
#include <esp_gap_ble_api.h>

BLEAddress LightSwitchAddr("84:2e:14:79:2f:f1"); //the address of the device I am connecting to

//the uuids of a characteristic/service I am trying to write to
BLEUUID ControlServiceUUID("720a9080-9c7d-11e5-a7e3-0002a5d5c51b"); 
BLEUUID OnOffCharacteristicUUID("720a9081-9c7d-11e5-a7e3-0002a5d5c51b");

BLEClient* client;
BLERemoteService* controlService;
BLERemoteCharacteristic* onOffCharacteristic;

constexpr auto LED_PIN = LED_BUILTIN;

void onOffStateNotified(BLERemoteCharacteristic* pBLERemoteCharacteristic, uint8_t* pData, size_t length, bool isNotify);

void setup() 
{
    Serial.begin(115200);
    pinMode(LED_PIN, OUTPUT);

    BLEDevice::setEncryptionLevel(esp_ble_sec_act_t::ESP_BLE_SEC_ENCRYPT_NO_MITM); //without this, the device doesn't even connect
    BLEDevice::init("esp32");

    client = BLEDevice::createClient();

    bool success = client->connect(LightSwitchAddr);

    if(success)
    {
        Serial.println("Connected");
    }
    else
    {
        Serial.println("Could not connect!");
    }

    controlService = client->getService(ControlServiceUUID);

    onOffCharacteristic = controlService->getCharacteristic(OnOffCharacteristicUUID);
    onOffCharacteristic->setAuth(esp_gatt_auth_req_t::ESP_GATT_AUTH_REQ_NONE); //not sure if this is necessary
    onOffCharacteristic->registerForNotify(onOffStateNotified); //this works fine
}

void onOffStateNotified(BLERemoteCharacteristic* pBLERemoteCharacteristic, uint8_t* pData, size_t length, bool isNotify)
{
    digitalWrite(LED_PIN, pData[0]);
}

long last = 0;
bool onOffState = false;

void loop() 
{
    if(millis() - last >= 1000)
    {
        last = millis();
        onOffCharacteristic->writeValue(onOffState = !onOffState, true);
    }
}