SoftwareSerial with receive timeout

How can I modify or what do I need to add or what do I need to modify on the SoftwareSerial sourcecode such that I will be able to determine the end of reception from the serial rx by some inter-character timeout?

What I mean is that when I receive a string like "1234567890", after about 100ms, a flag will be raised to indicate that new data has arrived and that the serial buffer read. I will just continously poll that flag to determine if I need to read the buffer. This is to avoid reading the serial buffer while in the middle of receiving serial data.

Isn't Serial.available() > 0 just that kind of "flag" ?

Nope it wont help so much. Serial.available() will always be raised when the characters 2,3,4, ..., 0 is still being received. If I poll serial.available() while the rest of the characters are being received and read the buffer, I wont be able to read the whole string.

The thing is, I dont know when the string will be received otherwise setting a simple delay will do the job.

Nope it wont help so much. Serial.available() will always be raised when the characters 2,3,4, ..., 0 is still being received.

Yeah, so? Serial.available() isn't a flag that gets raised. It returns the number of bytes available to read.

If I poll serial.available() while the rest of the characters are being received and read the buffer, I wont be able to read the whole string.

You seem to be looking for some magic way to know when all the serial data has been received. There is no such thing. The end of a packet of data is something YOU need to make sure is sent, and YOU need to read until that marker arrives.

PaulS:

Nope it wont help so much. Serial.available() will always be raised when the characters 2,3,4, ..., 0 is still being received.

Yeah, so? Serial.available() isn't a flag that gets raised. It returns the number of bytes available to read.

Sorry wrong choice of words on my part as I'm not a native english speaker. But the idea is still there that when you invoke Serial.available() while the sw serial port is still receiving data, you wont be able to get the whole data.

PaulS:

If I poll serial.available() while the rest of the characters are being received and read the buffer, I wont be able to read the whole string.

You seem to be looking for some magic way to know when all the serial data has been received. There is no such thing. The end of a packet of data is something YOU need to make sure is sent, and YOU need to read until that marker arrives.

Pardon my ignorance with AVRs and arduino .... I come from the PIC world. However what I want is NOT a magic way. The data I receive has no marker or any terminating characters and so my marker for the end of transmission is a timeout period and when that timeout occurs, a flag is raised that tells me its ready to read the buffer.

I simply want to be able to port my code originally written in C for the PIC mcu to the arduino platform. I have successfully ported my pic code to the AVR with AVR-GCC.

I'm just not that familiar yet with the interrupt driven s/w uart functionality of the SoftwareSerial library and how it actually works. Its still too cryptic for me to understand which is why I am asking for some help on where to start off with the code modification. I'm still new to arduino and the inner workings of the arduino core for me to do this.

But the idea is still there that when you invoke Serial.available() while the sw serial port is still receiving data, you wont be able to get the whole data.

Rubbish. There is no way to know when the serial port has finished receiving data, because that only makes sense in terms of one specific packet.

However what I want is NOT a magic way.

Yes, it is.

I simply want to be able to port my code originally written in C for the PIC mcu to the arduino platform.

We haven't seen any code, yet.

The serial port is an open ended connection. Data can arrive at any time. As each byte arrives, it is put in a buffer. Determine how much data is in the buffer, using Serial.available() (or the instance of SoftwareSerial's available() method). Read any buffered data using Serial.read() (or the equivalent software instance's read() method). Data can arrive while you are reading. That data will be put in the buffer, on the other end you are reading from.

There are no problems reading from the buffer while the instance is stuffing data in the other end.

This is a rough sketch that I wrote to assess the feasibility of using a timeout as EOT(*) marker.

(*) EOT = end of transmission.ù

Current timing is adequate for manual testing via serial monitor.

// serial timeout as eot marker

const unsigned int MAXBUF = 20;
char buffer[MAXBUF];
unsigned short bufCnt = 0;
const unsigned long eotTimeoutMs = 400;    // if no char received for 400 ms, declare end of transmission


void setup() {
    Serial.begin(9600);
}


void loop() {
    unsigned long lastCharTime = 0;
    
    if (Serial.available() > 0) {            // enter "receiving" state if something is waiting to be read()
        buffer[bufCnt++] = Serial.read();    // read first char
        lastCharTime = millis();             // timer start
        while ((Serial.available() > 0) || (millis() - lastCharTime < eotTimeoutMs) && (bufCnt < MAXBUF-1)) {
            if (Serial.available() > 0) {
                buffer[bufCnt++] = Serial.read();
                lastCharTime = millis();             // timer restart
            }
        }
        // exit from "receiving" state due to:
        // - timeout (proper eot marker)
        // - buffer full condition (TODO: should be reported as an error)
        
        // "close" buffer
        buffer[bufCnt] = 0;
        
        // do something with the message
        digitalWrite(13, HIGH);
        Serial.println(buffer);
        Serial.println(bufCnt);
        digitalWrite(13, LOW);

        // reset counter for next transmission        
        bufCnt = 0;
    }
}

PaulS:

But the idea is still there that when you invoke Serial.available() while the sw serial port is still receiving data, you wont be able to get the whole data.

Rubbish. There is no way to know when the serial port has finished receiving data, because that only makes sense in terms of one specific packet.

However what I want is NOT a magic way.

Yes, it is.

I simply want to be able to port my code originally written in C for the PIC mcu to the arduino platform.

We haven't seen any code, yet.

The serial port is an open ended connection. Data can arrive at any time. As each byte arrives, it is put in a buffer. Determine how much data is in the buffer, using Serial.available() (or the instance of SoftwareSerial's available() method). Read any buffered data using Serial.read() (or the equivalent software instance's read() method). Data can arrive while you are reading. That data will be put in the buffer, on the other end you are reading from.

There are no problems reading from the buffer while the instance is stuffing data in the other end.

Is it very hard to understand that you can determine the end of transmission of a particular string when after the last character of data is received, and no other character follows that after a specified time will already be indicative of end of string or transmission?

Let's put it this way ....

I send a string, say "1234567890" (with no LF or CR after the 0) at a constant baud rate. Between sending the characters 1 and 2, there is a certain time that elapsed, say 100usecs and this is constant between each characters betwen 2 and 3 and so on. So this goes on for the other characters until the character "0". After receiving the character "0", if after 100 usecs or more, there are no new characters received, then this would already be the marker that indicates that the whole string has been received. This way, any variable length string can be read w/o any end of string terminators.

The way I do this on the h/w uart port (interrupt driven serial reception) is when the interrupt happens as a new character is received, I set a h/w timer that will also interrupt when the set time has elapsed. This timer is always reset to 0 each time a new character is received within the timeout period. If the last character was received, no new character will reset timer and hence after the elapse time, it will cauase an interrupt. During this timer interrupt, I raise a flag to inidicate that all data from the unknown length string has been received. In the main loop, this flag is always polled, and when the buffer is read, the flag is cleared.

What I want is to be able to also do this on the SoftwareSerial uart.

If I do this with sofwareserial.available() while in the middle of receiving data, softwareserial would only return the partial numbers of characters and then I wont be able to capture the whole string.

mromani:
This is a rough sketch that I wrote to assess the feasibility of using a timeout as EOT(*) marker.

(*) EOT = end of transmission.ù

Current timing is adequate for manual testing via serial monitor.

// serial timeout as eot marker

const unsigned int MAXBUF = 20;
char buffer[MAXBUF];
unsigned short bufCnt = 0;
const unsigned long eotTimeoutMs = 400;    // if no char received for 400 ms, declare end of transmission

void setup() {
    Serial.begin(9600);
}

void loop() {
    unsigned long lastCharTime = 0;
   
    if (Serial.available() > 0) {            // enter "receiving" state if something is waiting to be read()
        buffer[bufCnt++] = Serial.read();    // read first char
        lastCharTime = millis();             // timer start
        while ((Serial.available() > 0) || (millis() - lastCharTime < eotTimeoutMs) && (bufCnt < MAXBUF-1)) {
            if (Serial.available() > 0) {
                buffer[bufCnt++] = Serial.read();
                lastCharTime = millis();             // timer restart
            }
        }
        // exit from "receiving" state due to:
        // - timeout (proper eot marker)
        // - buffer full condition (TODO: should be reported as an error)
       
        // "close" buffer
        buffer[bufCnt] = 0;
       
        // do something with the message
        digitalWrite(13, HIGH);
        Serial.println(buffer);
        Serial.println(bufCnt);
        digitalWrite(13, LOW);

// reset counter for next transmission       
        bufCnt = 0;
    }
}

Thanks so much, let me try to study your code and give it a try.

Glad to help. Feel free to ask.

Let's put it this way ....

I send a string, say "1234567890" (with no LF or CR after the 0) at a constant baud rate. Between sending the characters 1 and 2, there is a certain time that elapsed, say 100usecs and this is constant between each characters betwen 2 and 3 and so on.

There is a flaw in your logic. There is NOT a fixed duration between two characters at any given baud rate. The time between bits in a byte IS fixed, but the time between bytes depends on the load on the sender. If if is doing nothing but sending serial data, the time will be (relatively) uniform. If there are interrupts to be processed, the time between characters will not be uniform. YOU MUST NOT rely on any interval meaning that there is no more data in a packet.

The serial class does not KNOW that you are sending fixed size packets. Remember, this is the same hardware that uploads your sketch. Can you imagine if the uploader said "no, your code is too short. You must pad it with another umptitty ump million bytes"?

You should not rely on the code that mromari posted to know that you got a whole packet. That is not what that code is meant for. It IS meant to not block too long waiting for data that may never arrive.

Instead, you should use start and end markers to delimit your packets, and then read, and buffer, all data UNTIL the end of the packet arrives. Only then should you use the data. The arrival of an end of packet marker could be followed nanoseconds later by the start of the next packet. Or hours later. You never can tell.

Just wondering... If he controls the sender, he can have a certain level of control on timings, too, like how much time it will pass before a new packet is sent.

If he controls the sender, he can have a certain level of control on timings, too, like how much time it will pass before a new packet is sent.

Maybe, but relying on timing is a bad idea. Too many things affect timing. If you learn to write code that only handles little packets spaced well apart, then you will have trouble when the packets get larger, or the timing between packets is small or variable.

In my opinion (I'm starting to use that phrase more often here), it is best to use explicit start and end markers, and process data only when those markers arrive, storing or discarding everything else, as appropriate. Discard anything before a start marker, save everything up to the end marker.

This approach is not dependent on timing between packets/characters, and lets the Arduino do other stuff until a complete packet arrives.

When migrating from one platform to another, it is best not to try too hard to make the new platform do things the way the old one did them. Instead, look at the old code as a guide. Write new code for the new platform, doing things the new way, to achieve the same results, or as close as possible, as what was achieved on the old platform.

If OP will post the specific code that s/he is wanting to migrate, I'm sure we can discuss how the results can be achieved on the Arduino, without ruffling too many (more) feathers.

it is best to use explicit start and end markers

In fact in every project I've done so far I've always used this technique. I was just trying to follow what the OP was trying to say... having tried myself the timeout way in some experiments, I posted my prototype.

I have no control of the data I'm trying to receive, its just raw bytes w/o any headers or trailers or any start/end of text/data markers. The data is not periodic and the length of the data is variable.

I'll post my code later ... I'll simplify it as much as possible first but it is written for the microchip pic and uses a h/w uart instead of a s/w uart.

but it is written for the microchip pic and uses a h/w uart instead of a s/w uart.

So, why are you now trying to use software serial?