Thanks for the replies everyone. I will try and answer all the questions that were raised.
How do you identify the start and end of the data frame?
There is a start and stop bit. If you look at the second screen shot, the first and the last pulse shown are the start and stop bits (from what I can tell).
IF you measure the transmit pin with a volt meter when noting is being sent, you will see the quiescent voltage. When that changes, it will be the beginning of the "start bit". What is the quiescent voltage?
When I measure voltage on the sensor reply line at idle, I get 0 volts, when a bit is being sent it is at 5 volts.
Note the two traces are inverted from each other. Is the data being sent back as an RS232 signal, if so it needs level shifting and inverting before you can read it on an Arduino.
Those curser measurements are not one cycle but one and a half cycles. To measure one cycle always measure from one rising edge to the next, or one falling edge to the next.
The signal is not RS232 or RS485, it is TTL levels. I should have been more clear on the traces shown on the screen shot. The top trace is showing a different line than the sensor transmit line. It is showing an input to the sensor that acts as a request for the sensor to send a reply frame. I am zoomed way out on the first screen shot. So you can see that there is a request made by the controller (made by a pulse that lasts for about 7uS), then the sensor sends a reply frame (shown by the bottom trace). There are three requests in a group, the sensor replies to these requests at different speeds. The first two requests take a fair amount longer to reply to than the third one.
Yes, the cursors on the second screen shot actually have three bits between them. You may not be able to read it, but in the bottom right of the screen the time between the three bits is 325uS. I did this because the first bit on this frame measured a little less than most. This gives a bit width of about 108.3uS. But if I measure the full length of many frames, I get an average total length of 2.846mS. Dividing this by 26 (I have looked at enough frames to know with certainty that there are 26 bits) I get 109.46uS.
I believe it is a non standard type of frame because there is never consistently bits on in the same place except at the beginning and end of the 26 bit frame. If I ground out the sensing plate on the sensor, one of the replies will actually go to all zeroes except for the two bits on at the beginning (with one off in between) and one on at the end. Like this-
I have been able to figure out that the second and third bits have to do with the type of reply (moisture or temperature) and that every frame ends with a one. Also there are always 6 zeros after the start bit and the frame type. That leaves 16 bits for data. I think.
It should not be too difficult to write some code that works like regular serial detection code but which expects more bits. The code in Yet Another Software Serial might give you some ideas.
Note, however, that regular serial data relies on strict timing between the data bits.
Thanks for the link Robin2, I will check it out.