Serial transmission data loss [SOLVED]

Hi!

I am currently developping project to visualize signals comming from an encoder (ACP and ARP in the ocde hereafter).

Let me explain a bit:
For 1 rotation, there are 16384 ACP impulsions and 1 ARP impulsion. I have to store the length of ACP signals (period (10 bits) and highs (8bits)) and if there is an ARP (1bit). Then, I need 3 bytes to code 1 information.
I store these data with ISR (the chronometer is compatible with ISRs) and then I write via Serial in the main loop.

My problem is that after a period, I lose a certain number of buffers… To know this, I use CoolTerm software to save my data comming from the serial port.
With a Java application, I can visualize my data and then conclude that I have lost data.

I already tried a bunch of potential solutions :
-Writing on SD instead;
-Change buffer size;
-Change baudrate;
-Writing byte after byte instead of the whole buffer (ie for(j ; …) Serial.write(*(buff+j));

But it not concluant …

Can someone help me please to say why do I lose buffers ?

#include <eRCaGuy_Timer2_Counter.h>
#include <Wire.h>
#include <SPI.h>

//Constants
#define ACP_PIN                       20
#define ARP_PIN                       21
#define STOP_REC_BUTTON_PIN           40
#define BAUDRATE                      230400
#define BUF_SIZE                      600



//Variables
boolean         recordingEnded = false;
boolean         writing = false;
bool            arpNbr = 0;
unsigned short  acpHigh = 0;
unsigned short  acpPeriod = 0;
volatile int             i = 0;
char            buff1[3 * SIZE];
char            buff2[3 * SIZE];
volatile boolean         change_buffer = 0;


//Variables for interruption routines
volatile unsigned long acpHighUs = 0UL; volatile unsigned long acpPeriodUs0 = 0UL; volatile unsigned long acpPeriodUs1 = 0UL;

/**************************************************************************************/
/* Initialization                                                                     */
/**************************************************************************************/
void setup()
{
  Serial.begin(BAUDRATE);


  //Setup ACP and ARP inputs
  pinMode(ACP_PIN, INPUT_PULLUP);
  pinMode(ARP_PIN, INPUT_PULLUP);
  pinMode(STOP_REC_BUTTON_PIN, INPUT_PULLUP);

  //Setup the timer
  timer2.setup();

  //Attach interruption at the end of the initialisation (recording starts...)
  attachInterrupt(digitalPinToInterrupt(ACP_PIN), acpRising, RISING);
  attachInterrupt(digitalPinToInterrupt(ARP_PIN), arpRising, RISING);

}

/**************************************************************************************/
/* Main loop                                                                          */
/**************************************************************************************/
void loop()
{
  //Check if the user stops the recording
  if (digitalRead(STOP_REC_BUTTON_PIN) == LOW && !recordingEnded )
  {
    noInterrupts();
    delay(500);

  }
  
  //Check if the writing should be done
  if (writing && !recordingEnded)
  {
    if (!change_buffer) {
      Serial.write(buff1, sizeof(buff1));
    }
    
    else {
      Serial.write(buff2, sizeof(buff2));
    }

    writing = false;

  }

}

/**************************************************************************************/
/* This method is called at the beginning of each ACP.                                */
/**************************************************************************************/
void acpRising()
{
  digitalWrite(ACP_LED, HIGH);
  acpHighUs = timer2.get_count();
  acpPeriodUs1 = timer2.get_count() / 2 - acpPeriodUs0;
  acpPeriodUs0 = timer2.get_count() / 2;

  acpPeriod = acpPeriodUs1;

  attachInterrupt(digitalPinToInterrupt(ACP_PIN), acpFalling, FALLING); // To know when the signal goes down

}

/**************************************************************************************/
/* This method is called at the end of each ACP.                                      */
/**************************************************************************************/
void acpFalling()
{
  digitalWrite(ACP_LED, LOW);
  acpHighUs = (timer2.get_count() - acpHighUs) / 2;
  acpHigh = (unsigned short)acpHighUs;

  if (change_buffer) {
    buff1[i] = char((arpNbr << 7) +  (acpPeriod >> 8));
    i++;
    buff1[i] = char(acpHigh & 0xFF);
    i++;

    buff1[i] = char(acpPeriod & 0xFF);
    i++;

  }
  else {
    buff2[i] = char((arpNbr << 7) +  (acpPeriod >> 8));

    i++;
    buff2[i] = char(acpHigh & 0xFF);

    i++;
    buff2[i] = char(acpPeriod & 0xFF);

    i++;
  }

  arpNbr = 0;
  
  if (i == 3 * SIZE) {
    i = 0;
    change_buffer = !change_buffer;
    writing = true;
  }

  attachInterrupt(digitalPinToInterrupt(ACP_PIN), acpRising, RISING);
}

/**************************************************************************************/
/* This method is called at the beginning of each ARP.                                */
/**************************************************************************************/
void arpRising()
{
  arpNbr = 1;
}

UPDATE : It seems like, whatever the size of the buffer I use to store my data, I always loose some of them. I made few tests with 512x3 bytes buffers and I lost something like 3 to 8 buffers sometimes.

EDIT: It seems like it was CoolTerm the problem. I switched to RealTerm and I made a 1 night recording duration and I don’t have any errors!

Is there a reason for the 'i' variable to not be declared volatile?

Can your PC do a serial loopback test at 230400 without losing data? Some adaptors are better than other. It would be interesting to see what the upper limit is, and therefore how close you are currently are to it.

Yeah I probably missed it in my code but it is declared voltile. Have to change it ..

I made a bunch of different tests for example writing sentences throug Serial.println(...); and didn't have any losses with this...

I was wondering if the problem could come from storing data in ISRs so I switched to polling and I always have the same problem ...

Quite lost right now as I don't know what to do more and where does the error come from

Based on your information in you previous posts, ACP pulses can arrive every 244 microseconds which means that your rising and falling edge functions have to be completed well within 133 microseconds otherwise you will miss rising/falling edges due to the way you keep reassigning interrupts.

Wouldn't it be simpler to define all you interrupt handlers in setup?

I already checked ISRs duration. They take (both) around 25µ. So the problem shouldnt come from here, for me indeed.

Wouldn't it be simpler to define all you interrupt handlers in setup?

What do you mean ?

UPDATE : It seems like, whatever the size of the buffer I use to store my data, I always loose some of them. I made few tests with 512x3 bytes buffers and I lost something like 3 to 8 buffers sometimes.

How fast do your interrupts fill the buffer? So how often do you get those 512 bytes that you have to transmit? You will basically have to write the generated bytes faster than your interrupt is triggered.

In your other thread, Robin asked which board you're using and suggested that a Leonardo uses native USB and you can transfer at 480MBits / second (specified baudrate is irrelevant; there are other boards that can perform the same). Looking at the pin numbers that you use, you're currently more than likely on a Mega.

I also notice that you never set the writing variable to false; that might or might not be intentionally. As a result you keep pumping data as fast as the Arduino allows, even if a buffer is not filled yet with new data.

You are right for the writing boolean which never passes to false... Actually, I just cleaned up my code to make it more readable but I forgot some information... I changed my code so now it is correct.

How fast do your interrupts fill the buffer? So how often do you get those 512 bytes that you have to transmit? You will basically have to write the generated bytes faster than your interrupt is triggered.

Basiclly, this problem happens after like 6-10 minutes. DAta for the first (let's say) 50 turns are correct. By correct I mean that I don't have any losses.

In your other thread, Robin asked which board you're using and suggested that a Leonardo uses native USB and you can transfer at 480MBits / second (specified baudrate is irrelevant; there are other boards that can perform the same)

Do you think that Leonardo is better for serial writing than MEGA ? This could solve all my problems ...

Greg9901:
Basiclly, this problem happens after like 6-10 minutes. DAta for the first (let's say) 50 turns are correct. By correct I mean that I don't have any losses.

Is that 50 turns per second or 50 turns per hour or ...

Greg9901:
Do you think that Leonardo is better for serial writing than MEGA ? This could solve all my problems ...

Do the calculations :wink: I've never had a need for high speed but the proof of the pudding is in the eating.

Is that 50 turns per second or 50 turns per hour or ...

Nope. Actually there areseveral sepeed rotations and I am currently testing my code at 15 RPM. I wrote 50 turns just to give a range. The error occures between 50 and 100 turns actually.

I simply can't understand why do I loose data ... I checked every timing (for example if I do not write my buffer before it is filled) and everything is correct...

So you're getting 15 x 16384 interrupts per second? That would mean that you have 4 us to handle an interrupt.

You mentioned in an earlier post that the ISR takes 25 us; how do you match that against the 4 us?

I guess I'm missing something.

Note
minimum ISR duration is (from memory) about 8 us as registers need to be saved and later restored.

So you're getting 15 x 16384 interrupts per second? That would mean that you have 4 us to handle an interrupt.

Actually, 15 RPM stands for 15 Rotation Per Minute. So I have 15x16384x4 interrupts every minutes (x4 comes from HIGH and LOW interrupts for both ACP and ARP signals).

So according to you, the problem might come from the storing process in the ISRs ?

I also tested my algorithm with polling method insetead of ISRs and the problem remains the same... Notice that I wrote my bytes directly. I din't store them into a buffer.

Greg9901:
Actually, 15 RPM stands for 15 Rotation Per Minute. So I have 15x16384x4 interrupts every minutes (x4 comes from HIGH and LOW interrupts for both ACP and ARP signals).

As I said, “I guess I’m missing something?”. The M for minutes, not seconds.

Greg9901:
So according to you, the problem might come from the storing process in the ISRs ?

I think it’s the combination and serial is somehow the bottle neck.

Greg9901:
I also tested my algorithm with polling method insetead of ISRs and the problem remains the same… Notice that I wrote my bytes directly. I din’t store them into a buffer.

I wrote a small test sketch for a Leonardo and stored the results in a file on the PC using realterm. Each line is a two-digit number followed by . It ran for approximately 140 seconds and the file contained 1,699,856 records of 4 bytes. That indicates a throughput of about 388 kBits / second. I’m not sure why the advertised USB2 speeds are not reached.

But more importantly, I did not loose a record.

void setup()
{
  Serial.begin(57600);
  pinMode(LED_BUILTIN, OUTPUT);

  // wait for serial to become available
  while (!Serial)
  {
    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
    delay(200);
  }
  digitalWrite(LED_BUILTIN, HIGH);

  Serial.println("READY");

  // wait for a key
  while (Serial.available() == 0)
  {

  }
  digitalWrite(LED_BUILTIN, LOW);

}

void loop()
{
  static int cnt = 0;
  sprintf(buffer, "%02d", cnt);
  Serial.println(buffer);

  cnt++;
  if (cnt == 10)
  {
    cnt = 0;
  }
}

Actually, I don’t think that your example can show th problem I am facing. I totally agree with you that this type of example won’t present any mistakes. I made a bunch of them.

I think that the problem comes from the fact that I have to write a big amount of bytes over and over again but I can’t figure out why is it a problem …

But you used a Leonardo. I heard that there was a difference between Leonardo and mega or uno in serial communication. Is Leonardo better for this type of work?

Is Leonardo better for this type of work?

That is what I tried to show.

What I suspect that is happening is that due to slowish serial communication, you're overwriting your buffers. But I did forget to crunch your numbers to see if that is indeed the case.

I changed of software. I moved to RealTerm and it looks like my results are much better. I made a 1 hour recording and it peears that I had no errors... I will make a 1 night recording and see if errors still occure.

Otherwise, I will move to LEonardo as well to see if this is a "gamechanger" :slight_smile: