Sending Huge Data Over Serial Very Slow

Hello everyone,

I'm trying to send telemetry package and the GCS(Ground Station) checks all data reached or not,
if reached send command okay all of them reached and so on..

However, my other mission is sending 1 MB file from GCS to Arduino over serial.
When I'm trying to do this, it send 8000Byte in 950 ms. Baudrate is setted as 115200.

By the way I increased RX buffer size , I am sending 220 bytes at one time from GCS.

So all data(1MB file data) reached in 120 second and I think its very larger time than I expected is it?

Footnote : While sending 1Mb file data , arduino checks all data reached or not , and send GCS to package okey its reached send me another data and so on..

Show your code on both side.

Sending 1MB straight at 115200 bauds would take (approximately) less than 90 seconds.
Why don’t you bump the Serial speed to 1 or 2 million bauds?

With that volume of data, I’d suggest a checksum in your data blocks. Your protocol may already have that in place… just sayin’.

Thanks for reply @lastchancename and @J-M-L

I'll share my codes,

Im checking checksum like that:
Example :
< V C VID_LEN VIDEO_DATA >
V : VIDEO HEADER
C : COMMAND (COMMAND FOR ACTIVATE MOTOR AND ETC..)
VID_LEN : Checks VIDEO_DATA with strlen()
VIDEO_DATA : VIDEO BYTES
Example Data : <V 0 5 11111>

I can't increase Baud speed because I will use this system in Xbee (MAX BAUD 115200)
MAVLINKPROTOCOL.h (4.7 KB)
MAVLINKPROTOCOL.cpp (16.2 KB)

in "getDatas" , "waitforResponse" functions in .cpp file are doing whole work.

The example you’ve quoted has no checksum or error detection, other than the bytecount being correct

With such a large volume of serial data at a relatively high speed, there’s a good chance of bit errors over the span of the data over time.

If the error rate is low enough, the errors may not be discernible, but if you need ‘guaranteed good’ data, it’s mandatory to detect the error, flag it, and/or request a re-send of the faulty packet.

While it’s a different strategy, and uses longer runs of serial data (as an ASCIII format), take a look at the Intel HEX format for an idea… it provides an address (or packet number), a data length byte, followed by the data bytes - and a data checksum digit.
This allows the receiver to identify transmission errors, and request a re-send of the bad ‘address’ block.

The data could be binary, just that particular format uses a human readable hex numeric format for convenience.

If the data’s not critical, it can stop the transfer, ignore the error, or let the user know.
How you handle errors is up to you.

So, You are likely saying that change your protocol . So my question is this cause slowing down?
Because < V 0 5 11111 > is coming from GCS,
Arduino checks if the bytecounts Correct sends GCS to < V 1 > meaning that send me new bytes..
In this process, I don't miss any package and if the error happens sends again...

the count can be correct but one of the bit might be bogus. @lastchancename is saying use checksum or some other type of CRC

Okay @J-M-L I will use CRC.
How about the timing, what can you say about that? I mean why it's taking longer time than expected?

Am I doing something wrong?

Post code in line for both sides.

The overhead might just be in how you handle the check and also in wireless transmission (are you using ZigBee?)

For this test which takes 120 sec I didnt use Xbee, I used USB connection, I'm also sending my GCS code

Form1.zip (3.0 KB)
I have to upload .cs file as a zip because forum didnt allow it to upload.

Serial reading happening with DataRECV_EVENT function,
1Mb data sending happening with VIDEO_SEND_PROCESSING function.

Reading from my phone so can’t check zip… maybe tomorrow

You can just post your code in a reply instead of attaching it

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.IO.Ports;
using System.Diagnostics;
using System.Text.RegularExpressions;


namespace WinFormSerialing
{
    public partial class Form1 : Form
    {
        public String ReceivedString;
        public String ReceivedStringSAVED;
        public String LastSendedTelemetry;
        public char[] MARKERS = { '>', '<' ,'\n'};
        public Regex pattern = new Regex("[<>]");
        
        public String COMMAND = "0";
        public List<String> COMMAND_QUEUE = new List<String>();
        
        
        public static bool VIDEO_SIZE_SENDED = false;
        //public int VIDEO_SIZE = 10000; // 10 000 Byte
        public int VIDEO_SIZE = 1000000; // 10 000 Byte

        //public string VIDEO_BYTE = "11111"; // 5 Byte
        //public string VIDEO_BYTE = "1111111111"; // 5 Byte
        public string VIDEO_BYTE = "1111111111111111111111111111111111111111"; // 50 Byte
        //public string VIDEO_BYTE = "11111111111111111111111111111111111111111111111111111111111111111111111111111111"; // 80 Byte
        //public string VIDEO_BYTE = "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"; // 100 Byte
        public static bool VIDEO_SENDING_ACTIVE = false;
        public static bool VIDEO_SENDING_COMPLETED = false;

        public static bool SENDING_PERMISSION = false;
        public static bool MissingTelemetry = false;

        public static Stopwatch timePassedWatch = new Stopwatch();
        // Amaç , Data gonderdikten sonra veya command, bir yanıt gelmedi ise;
        // M C veya VS , veya V gonderdikten sonra ozellikle, Yanıt gelmediyse belirlenen sure içerisinde
        // Timer Tick etsin. ve datayı tekrar gondersin.

        public static int SENDED_BYTE = 0;
        public static int REACHED_BYTE = 0;

        public static int INCREASED_TIMES = 1;
        

        public string[] Splitted_Telemetry;
        public Form1()
        {
            InitializeComponent();
            Console.WriteLine(VIDEO_SIZE);
            string[] ports = SerialPort.GetPortNames();
            foreach (string port in ports)
            {
                PortLists.Items.Add(port);
            }
            SENDED_BYTE_LABEL.Text = SENDED_BYTE.ToString();
            REACHED_VIDEO_LABEL.Text = REACHED_BYTE.ToString();

            TextBox.CheckForIllegalCrossThreadCalls = false;
            
            

        }

       

        private void Connect_Event(object sender, EventArgs e)
        {
            serialPort1.PortName = PortLists.Text;
            //serialPort1.BaudRate = 9600;
            serialPort1.BaudRate = 115200;
            serialPort1.Open();
            System.Threading.Thread.Sleep(500);
            Console.WriteLine("Sending Start Byte!");
            serialPort1.Write("<S>");
        }

        public void setLastSendedTelemetry(String LASTWRITED)
        {
            LastSendedTelemetry = LASTWRITED;
        }

        private void DataRECV_EVENT(object sender, SerialDataReceivedEventArgs e)
        {

            ReceivedString = serialPort1.ReadLine();
            ReceivedString = ReceivedString.Replace("\n", string.Empty).Replace("\r", string.Empty); // Removed "\n" , "\r"
            if (ReceivedString.StartsWith("<")) // Telemetri)
            {
                ReceivedString = pattern.Replace(ReceivedString, ""); // Removed  "<" , ">" 
                Console.WriteLine($"Telemetri came {ReceivedString}");
                Splitted_Telemetry = ReceivedString.Split(',');
                
                if (SENDING_PERMISSION && MissingTelemetry) // SENDING PERMISSION VARSA INTERVAL COKTAN GELMIS DEMEKTIR
                                                                // VE HATALI PAKET GELDI KI MISSINGTELEMETRY OLMUS BU YUZDEN INVOK EDEBEILIRIZ.
                {
                    this.Invoke(new EventHandler(checkTelemetries));
                    //checkTelemetries();
                }
                //checkTelemetries();

            }
            else if (ReceivedString.StartsWith("I"))
            {
                
                Console.WriteLine($"Interval Came! {ReceivedString}");
                //this.Invoke(new EventHandler(SetInterval));
                System.Threading.Thread.Sleep(5);
                this.Invoke(new EventHandler(SetInterval));
                //SetInterval();
            }
            else if (ReceivedString.StartsWith("V"))
            {
                string[] splitted_STR = ReceivedString.Split(' ');
                if (splitted_STR[0] == "VS")
                {
                    Console.WriteLine($"VS BILGISI CAME {ReceivedString}");
                    String VS_REACHED = splitted_STR[1];
                    if (VS_REACHED == "1")
                    {
                        VIDEO_SIZE_SENDED = true; // SENDED SUCESSFULLY.
                        VIDEO_SENDING_ACTIVE = true;
                        timePassedWatch.Start();
                    }
                    //VIDEO_SEND_PROCESSING(); // IF SENDED OR NOT, DOESNT MATTER , GO TO THE FUNCTION,
                    // IF VS SENDED SEND VBINARY , IF NOT , SEND VS AGAIN.
                    this.Invoke(new EventHandler(VIDEO_SEND_PROCESSING));
                    //VIDEO_SEND_PROCESSING();
                }
                else if (splitted_STR[0] == "V")
                {
                    Console.WriteLine(ReceivedString);
                    
                    String V_REACHED = splitted_STR[1];
                    if (V_REACHED == "3")
                    {
                        VIDEO_SENDING_COMPLETED = true;
                        VIDEO_SENDING_ACTIVE = false;
                        REACHED_BYTE += VIDEO_BYTE.Length;
                        REACHED_VIDEO_LABEL.Text = REACHED_BYTE.ToString();
                        Console.WriteLine($"INCREASED TIMES _-------------> {INCREASED_TIMES}");
                        timePassedWatch.Stop();
                        Console.WriteLine($"TOTAL TIME TO REACH ALL BYTES {timePassedWatch.ElapsedMilliseconds}");
                    }
                    else if (!VIDEO_SENDING_COMPLETED)
                    {
                        if (V_REACHED == "1")
                        {
                            /* 
                                    INCREASE INDEX OF ARRAY. YENI STRINGE ATA ARRAYDAKI VALUEYI
                                    ORNEK :  array = [1,1,1,1,0,1]
                                    // indx = 40        
                                    // 111111111111111111111111
                                    // 000000000000000000000000
                            // array[0:first] , [first : first+40]
                                    VIDEO_BYTE =  array[FIRST_IDX : LAST_IDX].T oString() gibisinden..
                                */
                            int i = 0; // DELETE
                            INCREASED_TIMES += 1;
                            REACHED_BYTE += VIDEO_BYTE.Length;
                            REACHED_VIDEO_LABEL.Text = REACHED_BYTE.ToString();
                        }
                        else
                        {
                            Console.WriteLine("NOTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT");
                        }
                        // ELSE GEREK YOK ZATEN YUKARIDA IF'E GIREMDIGI ICIN INDEX FALN ARTMAYCAK.. OTOYINE
                        // AYNI VIDEO BYTE OLMUS OLACAK.
                        System.Threading.Thread.Sleep(5);
                        //VIDEO_SEND_PROCESSING();
                        this.Invoke(new EventHandler(VIDEO_SEND_PROCESSING));
                    }
                }
                
            }
            else if (ReceivedString.StartsWith("E"))
            {
                Console.WriteLine("------------COMM ENDED SIGNAL---------");
                SENDING_PERMISSION = false;
                MissingTelemetry = false;
                //PermissionTimer.Stop();
            }
            else
            {
                Console.WriteLine(ReceivedString);
            }
            
        }

        public void SetInterval(object sender, EventArgs e) // object sender , EventArgs e 
        {
            int INTERVAL = int.Parse(ReceivedString.Split(' ')[1].ToString());
            Console.WriteLine($"INTERVAL SETTED TO {INTERVAL-50}"); // INTERVAL -80
            PermissionTimer.Interval = INTERVAL-50;                 // INTERVAL -80
            PermissionTimer.Stop();
            PermissionTimer.Start();
            SENDING_PERMISSION = true;
            Console.WriteLine("PERMISSION IS GIVEN...");
            this.Invoke(new EventHandler(checkTelemetries));
            //checkTelemetries();
            //if (VIDEO_SENDING_ACTIVE && !VIDEO_SENDING_COMPLETED)
            //{
            //    this.Invoke(new EventHandler(VIDEO_SEND_PROCESSING));
            //}
        }
        public void checkTelemetries(object sender, EventArgs e) //object sender, EventArgs e
        {
            //ReceivedStringSAVED = ReceivedString; // WE COPIED IT. (Another Event can be trigger Thats why.)
            //string[] Splitted_Telemetry = ReceivedStringSAVED.Split(',');
            if (Splitted_Telemetry.Length != 11)
            {
                MissingTelemetry = true;
                //COMMAND_QUEUE_REQUEST();
                this.Invoke(new EventHandler(COMMAND_QUEUE_REQUEST));
                if (SENDING_PERMISSION) serialPort1.Write($"<M {COMMAND}>");
            }
            else
            {
                MissingTelemetry = false;
                TOTAL_PACK.Text = Splitted_Telemetry[1];
                TRANSFER_STATUS.Text = Splitted_Telemetry[10];
                Console.WriteLine("No missing Telemetries Checking videos and etc..");
                if (VIDEO_SENDING_ACTIVE && !VIDEO_SENDING_COMPLETED)
                {
                    Console.WriteLine("Videos IDENTIFIED ..");
                    //VIDEO_SEND_PROCESSING();
                    this.Invoke(new EventHandler(VIDEO_SEND_PROCESSING));
                }
                else
                {
                    if (SENDING_PERMISSION)
                    {
                        //COMMAND_QUEUE_REQUEST();
                        this.Invoke(new EventHandler(COMMAND_QUEUE_REQUEST));
                        serialPort1.Write($"<N {COMMAND}>");
                    }
                }
            }
            

        }

        private void Manual_Motor_Always_Click(object sender, EventArgs e)
        {
            COMMAND_QUEUE.Add("99");
        }

        public void COMMAND_QUEUE_REQUEST(object sender, EventArgs e) // NO EVENT INTHERE NORMALLY.
        {
            if (COMMAND_QUEUE.Count() == 0)
            {
                COMMAND = "0";
            }
            else
            {
                COMMAND = COMMAND_QUEUE.ElementAt(0);
                COMMAND_QUEUE.Remove(COMMAND_QUEUE.ElementAt(0));
            }
        }

        private void TenSecond_Active_Click(object sender, EventArgs e)
        {
            COMMAND_QUEUE.Add("77");
        }

        private void Manual_Release_Click(object sender, EventArgs e)
        {
            COMMAND_QUEUE.Add("88");
        }

        private void SEND_VIDEO_Click(object sender, EventArgs e)
        {
            VIDEO_SENDING_ACTIVE = true;
            //VIDEO_SEND_PROCESSING();
            this.Invoke(new EventHandler(VIDEO_SEND_PROCESSING));
            //VIDEO_SEND_PROCESSING();
        }
        public void VIDEO_SEND_PROCESSING(object sender, EventArgs e) // object sender, EventArgs e
        {
            //COMMAND_QUEUE_REQUEST();
            this.Invoke(new EventHandler(COMMAND_QUEUE_REQUEST));
            if (!VIDEO_SIZE_SENDED && SENDING_PERMISSION)
            {

                Console.WriteLine($"<VS {COMMAND} {VIDEO_SIZE.ToString().Length} {VIDEO_SIZE}>");
                serialPort1.Write($"<VS {COMMAND} {VIDEO_SIZE.ToString().Length} {VIDEO_SIZE}>");
            }
            else
            {
                
                if (!VIDEO_SENDING_COMPLETED && SENDING_PERMISSION && VIDEO_SENDING_ACTIVE) //VIDEO_SENDING_ACTIVE ADDED.
                {
                    
                    Console.WriteLine($"<V {COMMAND} {VIDEO_BYTE.Length} {VIDEO_BYTE}>");
                    serialPort1.Write($"<V {COMMAND} {VIDEO_BYTE.Length} {VIDEO_BYTE}>");
                    SENDED_BYTE += VIDEO_BYTE.Length;
                    SENDED_BYTE_LABEL.Text = SENDED_BYTE.ToString();
                }
            }
        }

        private void Permission_OVER_EVENT(object sender, EventArgs e)
        {
            Console.WriteLine("PERMİSSİON OVER.!!!!!!!!!!!!!!!!!!!");
            SENDING_PERMISSION = false;
            PermissionTimer.Stop();
            MissingTelemetry = false;
        }
    }
}

Did you calculate it? At 115200, one byte takes 86 microseconds. So a 1,000,000 bye file will take 86 seconds excluding possible overheads.

Yes, I tested it and I am saying it. 1Million Byte reaching times takes approximately 120 second.
I can also share whole .cs files (GCS application) you can try it yourself.

Here is My whole files as Zip
"WinFormSerialing" folder is belongs to c#
MAVLINKPRO.ino as you know
MAVLINKPROTOCOL.h and .cpp files .h are my protocol codes.

When you run c# program , You have to choose arduino port and when you click connect telemetry package will start to came.
Data packages(Telemetries) transmitted in 1Hz which means second by second.

To test how much second takes to reach 1MegaByte data , please click "Send Video" and watch Package label.
In other words, Package label represent 1 sec.

I could'nt upload my zip directly to this forum because its more than 4Mb (Approx 5Mb) , I upload into another website.

https://s6.dosya.tc/server3/92temf/WHOLE_CODE.zip.html

Have you considered using ethernet point to point using TCP-IP, you will be able to get a lot more speed.

but please consider range approx( max range 800m)

Which is it?
One megabyte or one megabit?

What link method are you using for that distance ?

MegaByte , I am using XBEE for handle range problems.