The question is why you have those 'long' delays? What are you trying to achieve with each of them? Do they have a function to prevent the screen from flickering? Do you want the servo to be in a position for a certain time?
You also have an array for toggleButton but you only have one button pin; what is the future plan?
Blink-without-delay is the wrong example to look at in my opinion. Look at the debounce example.
You have two ways to start the sorting (serial and button) but only one way to pause it (button); is this intentionally? What I would do is read the serial port and the buttons and keep a flag (you already do that for the button, re-use it for the serial port).
In a very simplistic form
void loop()
{
static int state = 0;
if(Serial.available() > 0)
{
Serial.read();
state = !state;
}
if(digitalRead(buttonPin) == HIGH)
{
state = !state;
}
if(state == 1)
{
// sort
}
}
It needs more than that; this is only to show how you can use one variable that is controlled by both the serial port and the button.