Volatile long changes too fast for Serial.Write

Hello,

For a project I am using an Arduino UNO with three RS422 shields on top of it. I use this arduino (and shields) for reading the output of an differential quadrature encoder. The generated data is sent from the arduino to a .NET application (C#) by the USB cable/serial port.

This all works fine when I use the Serial.print function to send the encoder value as a string to the serial port.

But this printing takes too much time, and is not very stable in time consuming. So I would like to use the serial.write function. This one is faster and is very stable.

But now a problem occured. There is an enourmous delay now (10-15 seconds) before the data is read in the .NET application.

I assume that this has to do with the fact that I use a volatile variable as encoder value, because the encoder value is changed in a intterupt function. I think that the value is changing while it is being written.

When I send a random long value to the serial port, there is no delay. When I add a delay after the writing of 10 ms there is also no big delay (it is when I use less than 10 ms delay). But that is far too much delay. The data has to be send as soon as possible.

Does anyone know how this delay occurs and are there solutions to fix this?

Here below the arduino code (the encoder_position is changing in the two interrupt functions):

#define encoder0PinA  3
#define encoder0PinB  2
volatile unsigned long encoder_position = 0;

void setup() {
  pinMode(encoder0PinA, INPUT);
  pinMode(encoder0PinB, INPUT);
  attachInterrupt(0, doEncoderA, CHANGE);
  attachInterrupt(1, doEncoderB, CHANGE);
  Serial.begin (115200);
  Serial.setTimeout(0);
}



void loop(){ 
   Serial.write((byte*)&encoder_position,4);
}

This is my C# code for reading the data:

 private void Port_DataReceived(object sender, SerialDataReceivedEventArgs e)
    {           
       if (Port.IsOpen)
        {          
            byte[] data = new byte[Port.BytesToRead];
            Port.Read(data, 0, data.Length);
            data.ToList().ForEach(b => recievedData.Enqueue(b));
            processData();
        }
    }
  private void processData()
    {
        if (recievedData.Count >= 4)
        {
            var packet = Enumerable.Range(0, 4).Select(i => recievedData.Dequeue());
            UInt32 pos = BitConverter.ToUInt32(packet.ToArray(), 0);
            Pulse = pos.ToString();
        }
    }

that's a shotgun approach! (and yes the value will change whilst being transmitted if an interrupt occurs)

how does your receiver know to read by chunks of 4 bytes (and how do you synchronise if you missed the first byte)

Standard procedure when accessing volatile variables (that exceed a single byte in size) outside the ISR is to disable interrupts, make a copy of the volatile variable, then re-enable interrupts. From then on, you can use the copy without having to worry about the value changing.

It is also possible to run serial communications at much higher baud rates than 115200 if the receiving end can handle it and the cable is fairly short. Not all that unusual to see someone's code running 1,000,000 baud.

J-M-L:
that's a shotgun approach! (and yes the value will change whilst being transmitted if an interrupt occurs)

how does your receiver know to read by chunks of 4 bytes (and how do you synchronise if you missed the first byte)

Why a shotgun approach? Did you see the processData() function in the C# code? I forgot to mention: When the Serial Port is opened, we clear the inbuffer.
When we send a normal (not volatile) long, it comes perfectly in the .NET application. So I assume the problem is not in the communication, but in the volatile-being of the variable.

david_2018:
Standard procedure when accessing volatile variables (that exceed a single byte in size) outside the ISR is to disable interrupts, make a copy of the volatile variable, then re-enable interrupts. From then on, you can use the copy without having to worry about the value changing.

It is also possible to run serial communications at much higher baud rates than 115200 if the receiving end can handle it and the cable is fairly short. Not all that unusual to see someone's code running 1,000,000 baud.

The interrupt pins must detect any changes on the encoder edges, because the encoder is running continuously and we can not miss any pulses. Nevertheless we tried to do what you suggest with using cli() and sei(). But this didn't change anything.

Funny detail: When we lower the baudrate to 9600 (so there is more time for the communication) the problem is gone. But on any higher baudrate, the problem occurs. But we need at least 115200

When writing data to the serial port without pause, then the TX buffer inside the Serial library gets full. It is 64 bytes. Once that is full, the Serial.println() or Serial.write() waits until a free spot comes available. That waiting means the sketch is doing nothing, just waiting. Because of that the sketch could become hundred times slower.

With 10 kbyte per second output, the speed of the Arduino Uno itself has influence. So you might indeed see a small difference between Wire.println() and Wire.write().

You should not have a Serial.write() free running in the loop() at maximum speed. That means you should fix your sketch to send only the information that is really needed. You have to re-think your project.

I think that you can not find a single example of a Serial output function that is free-running in the loop() without any constrains in the loop(), because it does not work.
Beside a full TX buffer in the Arduino Serial library, perhaps also buffers in the computer get filled.

The Serial.write() is not more stable than Serial.println(). They both work 100%. The Serial functions are not the problem.

When using a Leonardo/Micro or Zero/MKR board, then the USB device inside the processor is used. The baudrate is ignored and the data is transferred at USB speed. Even with extra software overhead, it is faster than the serial port of a Arduino Uno.

Why a shotgun approach

cf the free-running write() in the loop.

The point is that you spit out 4 bytes by 4 bytes (actually in practice byte by byte) with no separator. if your receiver gets out of sync (starts late for example) then you can't sync and have no way to know where a "frame" (4 legit bytes) really starts.

When I send a random long value to the serial port, there is no delay.

And what do you do with the encoder value in that case ? could it be that the the whole ISR gets optimized out ?
For any memory transfer to the UART interrupts anyway get disabled temporarily, so i don't see how the temporary disabling of interrupts to copy the 4 byte value could make you lose ticks anymore than what is happening already. But having said that, there is a way around it as well. You can make the counter 8-bit (volatile) of which you use only 7 bits, and where a change of state of bit 7 (the 8th bit) triggers an increment of an accumulating variable in the main program. In that way you can have a multi-byte value increment as long as the value is being read often enough for the change of the 8th bit to be registered.

Thank you all for your replies, they are very usefull and instructive.

I think I will just go back to my previous solution and just use the Serial.print.

Or will this also cause problems in the future with the TX Buffer?
Till now this worked for me all the time, so with the same code as in my post above, but using Serial.print in stead of Serial.Write.
But if this is tricky, I may have to consider to do it differently.

Serial.print() would send the ASCII representation of your data whereas Serial.write() sends the binary representation.

I suppose you have a different C# code for handling this ?

Yes, I have of course. I print the encoder value with a seperator char, so that I can read the ascii representation between the seperators. This works.

Hello,

Due to the issue above, I have an additinoal question:

I send the encoder values to the serial port with the Arduino UNO. I read it with a .NET application. The time between two readings is about 4ms and then I get multiple encoder values at the same time.
The serial printing takes 0.4 ms and the serial reading (.NET) takes max 0.1 s. So a time interval of 0.5-1 ms would be locical. But 4 ms is far too much.
I think this has something to do with the Serial send buffer of the Arduino. It looks like the arduino waits till there is a certain amount of data in the buffer before sending it. Is this true? And is there a way to make this time interval shorter. (I am sorry, but I am quite new with Arduino and serial communication)

Hope you can help me with this.

Riekelt

The serial printing takes 0.4 ms and the serial reading (.NET) takes max 0.1 s.

the printing and reading are asynchronous. Your .Net reads them as they come, can't read faster and when the last byte is sent a few nano seconds later your .NET app should have it.

. It looks like the arduino waits till there is a certain amount of data in the buffer before sending it. Is this true?

No. it sends the bytes as they come. But if you saturate the 64 bytes buffer, the printing request will become blocking until space is made available for your next print.

is there a way to make this time interval shorter

what happens if you set the baud rate to 2000000 (two millions bauds)?

I think this has something to do with the Serial send buffer of the Arduino. It looks like the arduino waits till there is a certain amount of data in the buffer before sending it. Is this true?

No i don't think so. Once the UART has sent the last 'bit' it reads a new byte from the buffer if available. That is i think it has 1 byte of extra hwBuffer, but almost no time is lost if that buffer is initially empty. This process is of course interrupt dependent, which means that it may be suspended while interrupts are disabled, which is while in an ISR.

what happens if you set the baud rate to 2000000 (two millions bauds)?

I did try it. Did not help.

This is what i am doing in .NET

using System;
using System.Collections.Generic;
using System.IO.Ports;
using System.Diagnostics;

namespace encoder_test
{
    class Program
    {     
        private static long count = 0;
        private static Stopwatch stop  = new Stopwatch();
        private static SerialPort port = new SerialPort("COM3", 115200, Parity.None,8, StopBits.One);
        private static string Data;

        static void Main(string[] args)
        {
            stop.Start();
            port.Open();
            List<long> timelist = new List<long>();
            List<string> encoderlist = new List<string>();

            while (count < 100000)
            {                           
                Data = port.ReadExisting();
               
                if (Data != "")
                {
                    timelist.Add(stop.ElapsedTicks);
                    encoderlist.Add(Data);            
                }
        
                count++;
            }
            foreach (var i in timelist)
            {
                int index = timelist.IndexOf(i);
                if (index > 0)
                {
                    long diff =  i - timelist[index-1];
                    Console.WriteLine("timestamp  " + i + "  enc val  " + encoderlist[index] + "  time diff  " + diff);
                }
            }
            Console.ReadLine();
        }
    }
}

This has the following outcome (console):

timestamp  4277599  enc val  536c82537c82538c82539c82540c82541  time diff  41055
timestamp  4319231  enc val  c82542c82543c82544c82545c82546c8254  time diff  41632
timestamp  4359453  enc val  7c82548c82549c82550c82551c82552c82  time diff  40222
timestamp  4400864  enc val  553c82554c82555c82556c82557c8255  time diff  41411
timestamp  4441641  enc val  8c82559c82560c82561c82562c82563c  time diff  40777
timestamp  4483070  enc val  82564c82565c82566c82567c82568c82569c  time diff  41429
timestamp  4523692  enc val  82570c82571c82572c82573c82574c82575c  time diff  40622
timestamp  4564713  enc val  82576c82577c82578c82579c82580c  time diff  41021
timestamp  4605595  enc val  82581c82582c82583c82584c82585c82586c  time diff  40882
timestamp  4647021  enc val  82587c82588c82589c82590c82591c82592c82  time diff  41426
timestamp  4687656  enc val  593c82594c82595c82596c82597c82598c82  time diff  40635
timestamp  4728677  enc val  599c82600c82601c82602c82603c82604c  time diff  41021
timestamp  4769319  enc val  82605c82606c82607c82608c82609c8261  time diff  40642
timestamp  4810776  enc val  0c82611c82612c82613c82614c82615c82616c  time diff  41457
timestamp  4851538  enc val  82617c82618c82619c82620c82621c82622c82  time diff  40762
timestamp  4892553  enc val  623c82624c82625c82626c82627c82628c8  time diff  41015
timestamp  4937633  enc val  2629c82630c82631c82632c82633c82634c8263  time diff  45080
timestamp  4974581  enc val  5c82636c82637c82638c82639c82640c82641c  time diff  36948
timestamp  5015383  enc val  82642c82643c82644c82645c82646c82647c  time diff  40802
timestamp  5056429  enc val  82648c82649c82650c82651c82652c82653c  time diff  41046
timestamp  5097141  enc val  82654c82655c82656c82657c82658c82659c826  time diff  40712
timestamp  5138688  enc val  60c82661c82662c82663c82664c82665c82666c  time diff  41547
timestamp  5179025  enc val  82667c82668c82669c82670c82671c82672c8  time diff  40337
timestamp  5220344  enc val  2673c82674c82675c82676c82677c82678c826  time diff  41319
timestamp  5261115  enc val  79c82680c82681c82682c82683c82684c82685  time diff  40771
timestamp  5301812  enc val  c82686c82687c82688c82689c82690c82691c  time diff  40697
timestamp  5343344  enc val  82692c82693c82694c82695c82696c82697c826  time diff  41532

As you can see. It takes 4 ms (40000 ticks at 10 GHz) before .NET gets a value, but when it gets one, it has multiple encodervalues ("c" is the seperator). There are a lot of empty reads between the readings with a value;

I am wondering why the serial port is empty for 4 ms and then get a lot of data at once.

The arduino is just doing:

void loop()
{   
  if(encoder_position != encoder_oldpos){

     Serial.print(encoder_position);
     Serial.print("c");
     encoder_oldpos = encoder_position;
  }

}

4 ms (40000 ticks at 10 GHz)

Hum, isn’t that 4 microseconds?