For the low pass filter with a cut off of 4Hz, would the low pass filter need to be an analog filter from the sensor to the arduino?
Yes.
It would also need to be a much better filter than a first order RC filter.
And could I simply add a delay(125) to set the code to sample at 8Hz?
In theory yes but only if all the other instructions took zero time to execute. You are better off with a timer calling an ISR and doing the sampling in the ISR.