xon/xoff flow control

Hi to everyone,
In a gps/gprs project, I'm trying to implement the xon/xoff flow control to increase the speed at which a file is transferred to a ftp server. I'm using an atmega 644p at 8mhz and with a serial speed of 38400. At 9600 the file uploads without problems (no flow control required).

My code is:

boolean send_data(){
  boolean end_file = false;
  char buffer;
  if (sd.begin(chipSelect, SPI_HALF_SPEED)){
    if (myfile.open(filename, O_READ)){
      altsoft.println ("sending data");
      buffer = myfile.read ();
      do{
        Serial1.print (buffer);	//Starts to send data
        if (Serial1.available ()){
          byte inchar = Serial1.read ();
          altsoft.println (inchar); //debug print
          if (inchar == XOFF){
            altsoft.println ("off"); //debug print
            do {
              inchar = Serial1.read ();
            } while (inchar != XON);
          delay (10);  
          }
        }
        buffer = myfile.read();
      } while(buffer >= 0);
      
      myfile.close ();
      Serial1.println ("+++");
    }
  }
 return (end_file); 
}

Although it's appears to detect xoff character there are missed characters (almost one line in each xoff) in the upload file. I attach part of the file where the characters are missing.

3898911,-692374,121001,202528,194,15,0,0
3898911,-692374,121001,202528,194,15,0,0
3898911,-692374,121001,202529,194,15,0,0
3898911,-692374,121001,202529,194,15,0,0
3898911,-692374,121001,202530,194,15,0,0
3898911,-692374,121001,202530,194,10,0         missing characters (one complete line)
3898911,-692374,121001,202531,194,15,0,0
3898910,-692374,121001,202532,194,15,0,0
3898910,-692374,121001,202532,194,15,0,0
3898910,-692374,121001,202533,194,15,0,0
3898910,-692374,121001,202533,194,15,0,0
3898910,-692374,121001,202534,194,15,0,0
3898910,-692374,121001,202534,194,15,0,0
3898910,-692374,121001,202535,194,15,0,0
3898910,-692374,121001,202535,194,15,0,0
3898910,-692374,121001,202536,194,15,0,0
3898910,-692374,121001,202536,194,15,0,0
3898910,-692375,121001,202537,194,15,0,0
3898910,-692375,121001,202537,194,15,0,0
3898910,-692375,121001,202538,194,15,0,0
3898910,-692375,121001,202538,194,15,0,0
3898910,-692375,121001,202539,194,15,0,0
3898910,-692375,121001,202539,194,15,0,0

I'm crazy trying to realize which could be the error but I cant detect it, I'll appreciate any help in this topic.

Thanks in advance

char buffer;

A buffer is generally a place to hold more than one of a thing. Naming the variable buffer when it can only hold one thing is not a good idea.

You seem to be sending data over a serial port, and expecting that the data WILL be delivered intact. The rules for serial data transmission were written by the United States Postal Service, which only guarantees to try to deliver your package, and only guarantees to try not to damage it.

There is no guarantee that the data will get there or that it will be good when it gets there.

Flow control at 38400 seems silly when you can bump the baud rate to almost three times that value.

Hi,
Thanks for the reply.
I supposse that some characters maybe lost using the serial communication but in every upload data is only lost in as many place as xoff's are send by the gsm module so I supposse that that data lost are related to my code. Due to the caracteristics of the gprs communication the data transfer could be variable so a flow control system should not be silly because gprs speed may vary from as few as 8 kbs to 80kbs.
Then, do you think lost data is not strage or not relating to the presented code?

but in every upload data is only lost in as many place as xoff's are send by the gsm module so I supposse that that data lost are related to my code.

I guess I missed the relationship between the xoff and the missing data.

Due to the caracteristics of the gprs communication the data transfer could be variable so a flow control system should not be silly because gprs speed may vary from as few as 8 kbs to 80kbs.

If you use the hardware serial port to talk to the GPRS, data is buffered, and the Arduino will simply feed it data as fast as it can, making the actual speed of the GPRS irrelevant.

If it were me, I'd be using an Arduino (or clone) that had more than one hardware serial port, like the Mega, the Micro, or the Leonardo.

I assume the data you're transferring consists of text and is known not to include any non-printing characters.

Your code looks like a reasonable implementation of soft flow control and I don't see any reason why it wouldn't work. My only slight doubt is that you send the first character before you check for flow control rather than the other way around, and only check for one XON/XOFF sequence at each character. It may be more robust if you changed the 'if (Serial1.available ())' into a 'while (Serial1.available ())' to ensure that any backlog of flow control characters had been processed before you send the next character, and do that check before sending the character rather than afterwards. I've no particular reason to suppose that will help at all, but would remove that type of thing as a concern.

To get positive proof that every character in the file was sent exactly once, you could write each char to altsoft at the point you write it to Serial1. (I assume that altsoft is at least as fast as Serial1.) Then you can compare the debug output from the sender with the uploaded file and confirm that the missing characters were in fact sent from your code.

If you have confirmed that everything was sent correctly then one possible explanation is that the flow control is happening in the wrong place. I don't know where the XON/XOFF characters come from, but for this scheme to work there would need to be a reliable communication path between the device that is sending the XON/XOFF, and your device which is receiving them. By reliable I mean that the path does not drop data without informing you that it has been dropped. If the path is not reliable then the device which is invoking flow control may continue to receive input (which it will eventually be forced to discard) which was sent before the XOFF reached the originator.

I guess I missed the relationship between the xoff and the missing data.

In the file uploaded to the ftp I find as many places with missed data as xoff's send the gprs module to the arduino. I have the gprs module connected both to a serial monitor and to the arduino and also the arduino to a serial monitor and to the gprs. In the attached code every time an xoff is detected by the arduino, the altsofserial send an off message so I can detect the xoff send by the module in the serial monitor and also the detected xoff by the arduino through the altsoftserial . So I dont know if the characters are lost just when the xoff is send by the module but it is at least curious that in every file I upload there are as many lines with missed characters as xoff sent by the gsm module. So I think this issue is related to the way I handle the xoff but I cannot detect what is wrong.

If you use the hardware serial port to talk to the GPRS, data is buffered, and the Arduino will simply feed it data as fast as it can, making the actual speed of the GPRS irrelevant.

You are right, data is buffered in the arduino and also in the gprs module (have a 2048 bytes buffer). But if I use a , for example 38400 bps speed to transfer to the module and the module send to the ftp the data at for example 9000 bps a buffer overflow could happen and in fact it happens as the gsm module send the xoff character to stop the data from the arduino.

PeterH I will try your approach, thanks!! But perhaps few things would change because the xoff is send by the module when its buffer still have free 63 bytes. Just now I realized that perhaps I detect the xoff character and stops the transmission from the sd card but the tx buffer of the hardware serial remains sending the characters stored and perhaps this is the cause of the missed characters. Perhaps the best solution would be to implement the xon/xoff flow control in the hardware serial library but I think that would be very difficult to my limited knowledge

PeterH I will try your approach, thanks!! But perhaps few things would change because the xoff is send by the module when its buffer still have free 63 bytes. Just now I realized that perhaps I detect the xoff character and stops the transmission from the sd card but the tx buffer of the hardware serial remains sending the characters stored and perhaps this is the cause of the missed characters. Perhaps the best solution would be to implement the xon/xoff flow control in the hardware serial library but I think that would be very difficult to my limited knowledge

I agree that this is what is at the root of the problem and would best (or can only) be addressed within the hardware serial library.

Lefty