Sending byte vector with high frequency

Sorry for opening two threads in little time, but they are part of a bigger project of mine, and I’m working on them in parallel.

Let’s suppose that I have a motor which runs for a little time every N seconds. That is simple to code. In parallel, I need to sample from an analog pin, with a frequency of at least 100 Hz.

This is the code:

void startSynchro(int motor_ind, int pulseT, long stepperdelay, int pulses, long interdelay)
{  
  short runsToStop = pulses;
  bool stepperUp = true;

  unsigned long currentMillis;
  unsigned long currentMicros;

  unsigned long previousMillisRead = 0;
  unsigned long previousMillisPulse = millis();
  unsigned long previousMicrosStepperPulse = 0;
  
  while (runsToStop > 0)
  {
    currentMillis = millis();
    int voltage;
    
    if (currentMillis - previousMillisRead >= SAMPLING_TIME)
    {
      previousMillisRead = currentMillis;
      voltage = analogRead(SAMPLING_PIN);
      /*send data to serial */
    }

    if (currentMillis - previousMillisPulse >= interdelay)
    {
      previousMillisPulse = currentMillis;
      triggerTime[pulses - runsToStop] = previousMillisPulse;
      for(unsigned long tStart = millis();  (millis()-tStart) < pulseT;)
      {
        currentMillis = millis();
        currentMicros = currentMillis*1000;
        if (currentMillis - previousMillisRead > SAMPLING_TIME)
        {
          previousMillisRead = currentMillis;
          voltage = analogRead(SAMPLING_PIN);
          /*send data to serial */
        }       
        if (currentMicros - previousMicrosStepperPulse >= stepperdelay)
        {
          previousMicrosStepperPulse = currentMicros;
          if (stepperUp)
          {
            digitalWrite(motor_ind + 1,HIGH);
          }
          else
          {
            digitalWrite(motor_ind + 1,LOW);
          }
          stepperUp = !stepperUp;
        }
      }
      stepperUp = LOW;
      runsToStop -= 1;
    }
  }
}

Basically is the BlinkWithoutDelay example extended to my scenario, with some more conditions because I must sample when my motor is both still both running.

As you can see there are comments, because whenever I sample, I must send timing and value to processing via serial. Since I think it’s hard for you (and it was for me too), to debug that code, I’ve written a firmware, with a similar logic, which I can debug more easily.

#include <FlexiTimer2.h>

byte TXBuff[8];

void myFunc()
{
  unsigned long t = millis();
  int r = 789; //random value, just to check if it's sent right
  noInterrupts();
  TXBuff[0] = 0xb3; // two header bytes
  TXBuff[1] = 0x3b;
  TXBuff[2] = t & 255;
  TXBuff[3] = (t >> 8) & 255;
  TXBuff[4] = (t >> 16) & 255;
  TXBuff[5] = (t >> 24) & 255;
  TXBuff[6] = r & 255;
  TXBuff[7] = (r >> 8) & 255;
  Serial.write(TXBuff,sizeof(TXBuff));
  interrupts();
}

void setup()
{
  Serial.begin(115200);

  FlexiTimer2::set(10,myFunc); //10 kHz samples
  FlexiTimer2::start();
}

void loop()
{
}

I was suggested by my professor (which worked with arduino and matlab, but I wanted to use processing), to use two header bytes in the packet to check if it is sent right, what do you think?

Processing code:

import processing.serial.*;

Serial port;

void setup()
{
  port = new Serial(this, "/dev/ttyACM0", 115200);
}

int previousMillis = 0;
void draw()
{
  if (port.available() > 0)
  {
    byte byteVec[] = new byte[8];
    byteVec = port.readBytes();
    if (((256 + byteVec[0]) == 0xb3) && (byteVec[1] == 0x3b))
    {
      int t = (byteVec[5] & 0xFF) << 24 | (byteVec[4] & 0xFF) << 16 | (byteVec[3] & 0xFF) << 8 | (byteVec[2] & 0xFF);
      println(t);
    }
  }
}

What happens is that printed times are not always 10 ms apart, but sometimes more (20 ms or 30 ms); moreover after some time I might get an ArrayIndexOutOfBoundsException exception.

I also tried with the APSync library, but sometimes I get delays of 20 ms too.

What do you suggest?

Your code is incomplete.

  unsigned long previousMillisRead = 0;
  unsigned long previousMillisPulse = millis();

Should these be static?

Frank-95:
I was suggested by my professor (which worked with arduino and matlab, but I wanted to use processing), to use two header bytes in the packet to check if it is sent right, what do you think?

I don't see any advantage in having two header bytes - it just makes finding them more complicated. IMHO a single start-marker and a single end-marker will give you a reliable system. Of course the start- and end-markers must be chosen so that they can never occur within the data. If you are sending data as text that is easy to achieve. If you are sending binary data it can be a bit more complicated. One possibility is to reserve the values 254 and 255 as markers. If that is not an option then set aside three byte byte values, say 253, 254 and 255 and use 253 as a special marker so that the receiver knows that the byte following it is NOT a marker. To send (for example) the byte value 254 you would send 253 254 etc.

What happens is that printed times are not always 10 ms apart, but sometimes more (20 ms or 30 ms); moreover after some time I might get an ArrayIndexOutOfBoundsException exception.

That level of timing precision with serial data may not be so easy to achieve. Try using a much higher baud rate - an Arduino Uno or Mega will work very nicely at 500,000 baud.

...R

I don’t see any advantage in having two header bytes - it just makes finding them more complicated. IMHO a single start-marker and a single end-marker will give you a reliable system. Of course the start- and end-markers must be chosen so that they can never occur within the data. If you are sending data as text that is easy to achieve. If you are sending binary data it can be a bit more complicated. One possibility is to reserve the values 254 and 255 as markers. If that is not an option then set aside three byte byte values, say 253, 254 and 255 and use 253 as a special marker so that the receiver knows that the byte following it is NOT a marker. To send (for example) the byte value 254 you would send 253 254 etc.

What is better between sending raw bytes or strings of integers in your opinion, in terms of speed and reliability?

Robin2:
That level of timing precision with serial data may not be so easy to achieve. Try using a much higher baud rate - an Arduino Uno or Mega will work very nicely at 500,000 baud.

Thanks but even with 500000 is not enough it seems. Strange because 500000 b/s = 500 b/ms = 62,5 B/ms, which is much higher than what I need, even if we consider computational cost at sender and receiver.

I changed the condition to check when it actually misses bytes (and switched from two header to one header and a tail byte):

    if (((256 + byteVec[0]) == 0xb3) && (byteVec[7] == 0x3b))
    {
      int t = (byteVec[4] & 0xFF) << 24 | (byteVec[3] & 0xFF) << 16 | (byteVec[2] & 0xFF) << 8 | (byteVec[1] & 0xFF);
      println(t);
    }
    else
    {
      print("missed");
    }

It almost never misses packets receiver side, because missing was being written rarely. On the other hand I understood that the problem is sender side because processing receive correctly the packets but with timestamps diferences of 10ms or 20ms, as if arduino missed the shipping of a packet

Frank-95:
On the other hand I understood that the problem is sender side because processing receive correctly the packets but with timestamps diferences of 10ms or 20ms, as if arduino missed the shipping of a packet

Maybe the problem is with the FlexiTimer2 library. Why not just write your own timing code in loop() using millis() or micros(). Something like

if (micros() - prevSendMicros >= 10000) {
   prevSendMicros = micros();
   // code to do the sending
}

...R

You read too many millis() in a single run. That is not advised. Maybe try to restructure your code to just read one instance for each run.

Robin2:
Maybe the problem is with the FlexiTimer2 library. Why not just write your own timing code in loop() using millis() or micros(). Something like

if (micros() - prevSendMicros >= 10000) {

prevSendMicros = micros();
  // code to do the sending
}




...R

It doesn't change anything. Anyway, I tried to print Strings and read them upon the Arduino Serial Monitor and they work

void myfunc(unsigned long t)
{
  noInterrupts();
  Serial.println(t);
  interrupts();
}

void setup()
{
  Serial.begin(500000);
}

unsigned long previousMillis = 0;
unsigned long currentMillis;
void loop()
{
  currentMillis = millis();
  if ((currentMillis - previousMillis) > 0)
  {
    myfunc(currentMillis);
    previousMillis = currentMillis;
  }
}

I can clearly read the different millis 1 ms away from each other, but the problem arises when they are read via processing.

arduino_new:
You read too many millis() in a single run. That is not advised. Maybe try to restructure your code to just read one instance for each run.

Do you mean the first code? the thing is that I must keep on sampling when both when the motor is sill both when it's running, so I need to call millis both inside the while, both inside the for loops

BlinkWithoutDelay does not use while() or for(). You should not use them either.

The clue is your method forces you to copy-paste code. If you don't use while() or for() then there is no copy-paste.

MorganS:
BlinkWithoutDelay does not use while() or for(). You should not use them either.

The clue is your method forces you to copy-paste code. If you don't use while() or for() then there is no copy-paste.

The blink without delay does not use while or for because it is inside the loop which it's not possible in my case. The loop is substituted by the while (because it must stop after some condition), and the for statement is to run the motor for a certain amount of time. The while condition is given by the number of motor pulses executed, because the for inside the run is not always executed as you can see but only at some time intervals, and it does work

Frank-95:
The blink without delay does not use while or for because it is inside the loop which it's not possible in my case.

That is exceedingly unlikely. The whole purpose of loop() is to repeat things.

Have a look at how the code is organized in Several Things at a Time

Note how each function runs very briefly and returns to loop() so the next one can be called. None of the functions tries to complete a task in one call. And there may be dozens of calls to a function before it is actually time for it to do anything.

...R

Okay thanks, I’ll see how I’ll manage to rewrite the code.

Anyway, it seems to me that you are talking about the very first code, the one that runs the motor and samples, am I right?
But as you’ve seen the problem doesn’t lie in that code, but in the communication between arduino and processing; in fact, if we consider simpler cases like the ones I posted after that, the problem still persists.

This means, that even if I changed the code as you suggested me, it wouldn’t solve the problem.

This is the minimal example I already posted, with the changes you suggested (not using FlexiTimer2):

byte TXBuff[8];

void myFunc(unsigned long t)
{
  noInterrupts(); //doesn't change anything if it is commented
  TXBuff[0] = 0xb3;
  TXBuff[1] = t & 255;
  TXBuff[2] = (t >> 8) & 255;
  TXBuff[3] = (t >> 16) & 255;
  TXBuff[4] = (t >> 24) & 255;
  TXBuff[5] = 0x00; //empty for now
  TXBuff[6] = 0x00;
  TXBuff[7] = 0x3b;
  Serial.write(TXBuff,sizeof(TXBuff));
  interrupts();
}

void setup()
{
  Serial.begin(500000);
}

unsigned long previousMillis = 0;
unsigned long currentMillis;
void loop()
{
  currentMillis = millis();
  if ((currentMillis - previousMillis) > 9)
  {
    myFunc(currentMillis);
    previousMillis = currentMillis;
  }
}

And this is the processing code:

import processing.serial.*;

Serial port;

void setup()
{
  port = new Serial(this, "/dev/ttyACM0", 500000);
}

byte byteVec[] = new byte[8];
void draw()
{
 if (port.available() > 0)
 {
   byteVec = port.readBytes();
   if (((256 + byteVec[0]) == 0xb3) && (byteVec[7] == 0x3b))
   {
     int t = (byteVec[4] & 0xFF) << 24 | (byteVec[3] & 0xFF) << 16 | (byteVec[2] & 0xFF) << 8 | (byteVec[1] & 0xFF);
     println(t);
   }
 }
}

What happens is that printed times are spaced 10/20/30 ms apart, while they should be always 10 ms apart; it’s like if arduino missed to send some packets.
This is strange because if I didn’t use processing to read packets, but just the Arduino IDE serial monitor with this code

void myfunc(unsigned long t)
{
  noInterrupts();
  Serial.println(t);
  interrupts();
}

I’d just get the right numbers, even if I increase the sampling from 100Hz to 1kHz.

This is what I cannot understand. The first code might be written better, but I wrote it there to give you some context, the loops just work good. It’s sending packets (that must be done inside there) the problem

Frank-95:
Anyway, you're talking about the very first code which is the final step, but as you can see the problem arises also during simpler scenarios that I'd like to make them work before getting to the harder one. Also because I can also rewrite the code as you say, but if it doesn't work when it just has to send bytes and nothing else, how could it work with all that extra stuff? Am I not right?

Please read that out loud to yourself and see if you can understand it. I can't.

And surely the final step is always the very last code, not the first?

...R

Please read that out loud to yourself and see if you can understand it. I can't.

I've edit the post

Making big changes to an earlier Post is very confusing - I have no idea what was in the original and what is new. Please put new stuff in a new Reply in chronological order.

I'm not sure that your editing has much clarity either. For example

Anyway, it seems to me that you are talking about the very first code

who do you mean by "you" and in which Post is the first code to be found?

Looking at the minimal example code in Reply #10 I don't understand why you have noInterrupts() as there does not seem to be any interrupt service routine (ISR).

Serial won't work when interrupts are disabled.

...R

who do you mean by "you" and in which Post is the first code to be found?

I've been told to not use while and for, which was clearly referring to the first code. I don't know how else I could say the first code, the code in the first post :stuck_out_tongue: The startSynchro function

MorganS:
BlinkWithoutDelay does not use while() or for(). You should not use them either.

The clue is your method forces you to copy-paste code. If you don't use while() or for() then there is no copy-paste.

Then I reply and you suggested me how to change that function. In post #10, I just edit what you didn't understand to explain myself better.

Anyway, back to the topic.

Robin2:
Looking at the minimal example code in Reply #10 I don't understand why you have noInterrupts() as there does not seem to be any interrupt service routine (ISR).

Serial won't work when interrupts are disabled.

I commented those functions, but the result doesn't change (see post #10)

Anyway I found a reply of yours in a similar topic, I'll see if I can do it like that

You probably think I sound like a real PITA but, the only knowledge I have about your problem is what you tell us. And keep in mind that I am not focused on your problem in the way that you are. I look at maybe a dozen or two dozen problems every day and when I come back to see your next Reply I will probably have forgotten most of the stuff you wrote earlier. Hence naming the people you refer to and mentioning the number of the Reply in which something is to be found makes it quick or me to catch up.

I commented those functions, but the result doesn't change (see post #10)

And you don't seem to have taken notice of what I said about not changing earlier replies. That does not encourage me to help you.

Collecting an array of data and then sending it at leisure may be a solution to your problem, but I had assumed from your Original Post that you need real-time data delivered to your PC.

...R