Nicla vision RPC programming get effected by power supply

Hardware Nicla family Nicla Vision
1

Core M7 needs data returned from Core M4. I have managed to achieve that perfectly. However, that only works fine in cause I use Power Supply from micro USB (connect to my computer). When I use the other board to supply 5v via the J2 VIN, then the core M4 gets latency issue, it falls behind the core M7 about 1-2 seconds.

This is my code

#ifdef CORE_M7
void remote_update(int param1) {
  if (param1 < 1) digitalWrite(LEDR, LOW);
  else digitalWrite(LEDR, HIGH);
}
#endif

void setup() {
  RPC.begin();
  
#ifdef CORE_M7
  RPC.bind("remote_update", remote_update);
#endif
}

void loop() {
#ifdef CORE_M4
  int param1 = ...;
  RPC.call("remote_update", param1);
  delay(50);
#endif
}

Any idea? please discuss!