I observe this too. If there is no connection the numbers are randomly.
If the Tx/Rx are connected the numbers count up 48,49 etc.
Rx/Tx connected
09:09:57.519 -> "received" byteFromSerial=48
09:09:58.502 -> "received" byteFromSerial=49
09:09:59.521 -> "received" byteFromSerial=50
09:10:00.506 -> "received" byteFromSerial=51
09:10:01.499 -> "received" byteFromSerial=52
09:10:02.513 -> "received" byteFromSerial=53
09:10:03.499 -> "received" byteFromSerial=54
09:10:04.509 -> "received" byteFromSerial=55
09:10:05.525 -> "received" byteFromSerial=56
09:10:06.508 -> "received" byteFromSerial=57
09:10:07.527 -> "received" byteFromSerial=58
09:10:08.508 -> "received" byteFromSerial=59
09:10:09.529 -> "received" byteFromSerial=48
09:10:10.509 -> "received" byteFromSerial=49
DIS-connected
09:10:39.990 -> "received" byteFromSerial=224
09:10:39.990 -> "received" byteFromSerial=0
09:10:39.990 -> "received" byteFromSerial=0
09:10:40.024 -> "received" byteFromSerial=0
09:10:40.024 -> "received" byteFromSerial=0
For connecting a 5V-Arduino to a 3.3V ESP32 you should use voltage-level-converters.
For Arduino sends to ESP32 a simple voltage-divider would be sufficient.
For sending from the ESP32 to an 5V arduino voltage-level-shifting is nescessary.
best regards Stefan
For sending data from an arduino to an ESP32 over serial I recommend reading
best regards Stefan