You already noticed that loop will be very often, and you correctly modify brightness step by step from one loop run to the next.
Similar, you switch direction
fadeAmount = -fadeAmount;
at the end.
But then you destroy that logic by
//delay(100);
brightness = 0;
fadeAmount = 10;
... if I get your code right.
After you got that fading right, you should add another state variable ( like your brightness, which keeps a state from one loop to the next ) telling where in the overall story you are.
This might be a "step" going from 0 to 23 and being used as an index to your array of duration times.
I hope this delay per loop should control the fading speed ?