You have two main problems in the sketch:
-
You have no delimiter character. So the only way to recognize the beginning of the number string on the serial interface is a short pause but by removing the delay you don't even have that anymore. So define a delimiter code (carriage return might be a good choice) and start a new number if you receive that character.
-
You're using the String class. That might be responsible for the freeze you see after a few minutes. That class is not designed for embedded computers like the Arduino because it fragments the memory and soon enough you'll end with no large enough block available. The Arduino then reacts unpredictable, most often it's simply freezing. As you don't use the constructed string other than returning it to the serial interface, you can delete that completely and return the contents of your numbers array.