Capturing a digital pulse mixed in an analog signal

I'm looking to capture a digital pulse (read via an IR phototransistor on an analog input), and filter out the noise while automatically calculating and continually adjusting the trigger point. My digital pulse is about 30ms, surrounded by long periods (600-160000+ms) of silence. It seems like I can capture from 2 to 11 ADC samples during that high pulse in my current sketch, which includes sending data out the serial port and updating an LCD display every 10 samples. The important variable here is the interval between each pulse.

In my current sketch, I started out reading the analog value and adding it to an array of 25 int as a circular buffer, which I sort to calculate the median and mean values of the array. My "noise" currently varies from 0 at night to about 50 ADC counts over the last day and a half, but I'd like to do my best to make sure I don't have to get in and tweak the sketch in the summer just because the noise got a little higher than a hard-coded trigger value. My peak ADC value for the digital high period has been as high as 800-900, but seems more towards an average of 300-400 adc counts during the period of samples where a high was captured. A friend suggested I do a FIR or FFT on the data, but that seems more geared towards a consistently repeating frequency rather than a pulse that could have a random delay between each period. Would it be to my benefit to use one of those filters or would I be fine to just calculate a standard deviation on the samples collected? Is my buffer large enough or should I double it?

Once I have the information, my thought was to normalize the noise down to zero and scale the peaks to max, and then pick a number somewhere in the upper-middle as the transition level.. thus, hopefully maintaining a healthy response until the noise completely drowns out the signal.

My digital pulse is about 30ms, surrounded by long periods (600-160000+ms) of silence. It seems like I can capture from 2 to 11 ADC samples during that high pulse in my current sketch, which includes sending data out the serial port and updating an LCD display every 10 samples.

In order to find something, you need to know what to look for. About 30 ms ? That all you know? Well it still would be possible to filter, but more specific on pattern 'd be helpful. Forget FIR and FFT, IIR. Digital signal (same as visual) has sharp edges, though median filter would be the best, IMHO. When you say you try, do you have a link or code? And last things, you need exactly 10 samples per 30 msec, no more no less. LCD and serial shouldn't interfere.

in my current sketch, it's running full speed, pulling samples and doing median as well as mean calculations. when I feed a constant stream of 25 samples and then dump it to serial, I got an estimated 2-3 Ms between samples. about 30Ms pulse time is all I could find for info, supported mostly by my sample rate which indicated about 26Ms when looking at the high periods logged. something like 65Ms to process those 25 samples. don't have access to my code, but will try to get a snippet up later. when I looked up info on standard deviation, it looks like I would be doing a lot of squares and square root functions.. and I was a bit worried whether I could do all that work in enough time to keep the sample rate up. thinking about it this morning, I thought about Using standard deviation to separate the highs from the lows, then processing standard deviation on the high periods alone, then scaling the data with a slope function and seeing the trigger levels at 30% and 70% of those values accordingly. sound like a solid plan?

Sorry, nothing of above does make any sense to me. I'd refer to this book (my favorite, btw): http://101science.com/dsp.htm and especially this chapter: http://www.dspguide.com/ch15/2.htm

So you're searching for the equivalent of a click on a phone line?

Close enough analogy.. minus the dialtone.

If it's a louder click than the ambient signal, you could watch for high amplitude.

You can surely estimate the background level with a low-pass filter that will ignore the 30ms pulses
(time constant of 1s or so?), then estimate the noise level by calculating the rms of the ac component
across the sample buffer.

Set a detection threshold at say 5 to 10 sigma above the background level?

As has been pointed out you can also band-limit the signal to reduce noise amplitude (a simple RC filter
with time constant of 2 to 5ms or so?)

Is sigma the same as Standard Deviation? (I think the answer is yes..) Never took calculus and struggling to wrap my head around bits of it as I dig around online.

The noise is more of a slowly increasing DC offset rather than a random distribution. My current logged dataset revolved around an arbitrarily picked midpoint and ignored logging anything below that point. Once the signal exceeded that midpoint and subsequently dropped below it again, I logged the 25 previous samples. When I did a histogram on that cumulative bit of data, I found 2 peaks.. The main one being the low (with some 45k+ points) and the other surrounding the average high point (with a peak of about 5k points). So it seems like if I create a slowly periodic histogram with a slow decay(to cast out the outliers), my controller can identify and focus on those two peaks, get a mean value for each and use that to scale the input and set my threshold. If I use standard deviations on those two peaks separately, I can minimize the need to frequently do square root calculations and keep my sample rate and accuracy high.

dnear1:
Is sigma the same as Standard Deviation? (I think the answer is yes…) Never took calculus and struggling to wrap my head around bits of it as I dig around online.

Yes, noise is randomly distributed so its rms amplitude is 1 sigma by definition. 5 sigma is good enough for CERN I believe. What’s calculus got to do with it?

Because if you could -make- the clicks in the input stream very loud, they would be easy to spot. For instance, my cheap ultrasonic sensor works even with music playing.