Send string serial data from arduino to processing

Projects Programming
1

Hi I'm trying to send a float array from arduino to processing as a string with commas separating the variables. I know how to receive the data and split it. My reasoning is that I am already sending serial data from my arduino using the Serial.print and Serial.println but this is slowing down my program extremely which is running a stepper motor and I can't attain the speeds I want when using the "print" statements, so I was looking into using serial.write instead. I don't want to use a for loop to send them as bytes because the order keeps getting messed up for some reason.

i am sending three dynamic variables variables(sensor data) ranging from 30 -1200 and I basically want to send it as, example ("x-val,y_val,z_val") and split the string values into an array which I can already do.

I looked into the sprintf but for some reason, I'm not getting it

2

You can do: Serial.write(buf, len);

buf being an array of bytes and len being the number of elements you want to print out of buf.

The big thing you need to worry about, though, is you are actually trying to transfer a series of 16-bit numbers since the highest value you need to represent is 1200. The max you can get with a byte is the number 255. I suggest the following:

void sendData()
{
  byte buf[] = {'<', 0, 0, 0, 0, 0, 0, '>'};
  byte len = 8;

  buff[1] = (x_val >> 8) & 0xFF;  //MSB
  buff[2] = (x_val) & 0xFF;       //LSB

  buff[3] = (y_val >> 8) & 0xFF;  //MSB
  buff[4] = (y_val) & 0xFF;       //LSB

  buff[5] = (z_val >> 8) & 0xFF;  //MSB
  buff[6] = (z_val) & 0xFF;       //LSB

  Serial.write(buf, len);
  
  return;
}

You can also do Serial.write(String); but you'll get faster results with the above method.

3

You can send binary data about twice as fast as text data. Depending on how much extra commas and whitespace were in the text data, maybe 3 or 4 times faster.

Is 2-4 times faster fast enough? I'm guessing that it isn't. You need to change your system. Instead of sending the entire array on every step of the motor, send it after every 10 steps or every 100 milliseconds.

4

Also, might want to jack up the baud to 115200 if you're not already using it. Be aware too that the real bottleneck could be the sensors and not the serial stream. Some sensors can take a while (in microcontroller time) to report their data.

5

Power_Broker:
You can do: Serial.write(buf, len);

buf being an array of bytes and len being the number of elements you want to print out of buf.

The big thing you need to worry about, though, is you are actually trying to transfer a series of 16-bit numbers since the highest value you need to represent is 1200. The max you can get with a byte is the number 255. I suggest the following:

void sendData()

{
  byte buf[] = {'<', 0, 0, 0, 0, 0, 0, '>'};
  byte len = 8;

buff[1] = (x_val >> 8) & 0xFF;  //MSB
  buff[2] = (x_val) & 0xFF;      //LSB

buff[3] = (y_val >> 8) & 0xFF;  //MSB
  buff[4] = (y_val) & 0xFF;      //LSB

buff[5] = (z_val >> 8) & 0xFF;  //MSB
  buff[6] = (z_val) & 0xFF;      //LSB

Serial.write(buf, len);
 
  return;
}




You can also do Serial.write(String); but you'll get faster results with the above method.

I'm already using 115200 baud rate too and I've tested the sensor data, everything works fine without the Serial.println slowing it down. And unfortunately I need the array at every step of the motor to give the positional data.
I tried this code and I receive it in processing but can't really interprete the data
my processing code is below

import processing.serial.*;
Serial myPort;
void setup()
{
  String portName="COM3";
  myPort = new Serial(this, portName, 115200);
}

 
void draw()
{
  
  
 
  if (myPort.available() > 0) 
  {
     byte[] inbuffer = new byte[8];
myPort.readBytesUntil('>',inbuffer);
      String mystring = new String(inbuffer);
      println(mystring);
 

  }
  
}

picture showing how the data is received is also attached

jjj.PNG
jjj.PNG1298×901 29.1 KB

6

X, Y and Z all change on every step? Do they change in a predictable way? Like the Arduino can tell Processing where it is going and then send a single byte to say a step is complete?

Why does Processing need the real-time data? Could you record the data and send it afterwards?

If you use an Arduino with Native USB then you can get speeds a lot higher than 115200. Even the regular ones can go up to a million or two.

7

kikisarpong:
Hi I'm trying to send a float array from arduino to processing as a string with commas separating the variables. I know how to receive the data and split it. My reasoning is that I am already sending serial data from my arduino using the Serial.print and Serial.println but this is slowing down my program extremely which is running a stepper motor and I can't attain the speeds I want when using the "print" statements, so I was looking into using serial.write instead.

Another option is not to send the entire message in one go. The receiving program won't be able to tell the difference between these two code snippets

Serial.print("<Hello world>";

Serial.print("<Hello ");
Serial.print("world>");

And you should be able to use 500,000 baud

...R

8

MorganS:
X, Y and Z all change on every step? Do they change in a predictable way? Like the Arduino can tell Processing where it is going and then send a single byte to say a step is complete?

Why does Processing need the real-time data? Could you record the data and send it afterwards?

If you use an Arduino with Native USB then you can get speeds a lot higher than 115200. Even the regular ones can go up to a million or two.

I haven't tried any baudrate above 115200..well I think I tried 500000 once and my program wasn't working but I didn't really troubleshoot to see if anything else might have been the problem so I'm not sure
and yes my X,Y,Z data changes on every step and I do need real time data-- I'm building a 3D mapping system and i'm done with the mapping environment and everything else. well, about 70% maybe but I'm trying to get my stepper motor to above 1000 rpm and anytime I serial transfer the data, it slows down.
Any serial.print in the code slows the motor.

I tried this code and its not really helping either

  char st[20];
 float arr[] = {x_val,y_val};
 String str =  String(arr[0]) +  "," + arr[1]  + "|";
 str.toCharArray(st,sizeof(st));                    
  Serial.write(st)

and I split the string in processing

9

Stay away from big-S strings. That one line of code creates a lot of work for the Arduino. It also carries a risk of memory fragmentation and random failures due to running out of memory.

So let's strip it down to the absolute smallest possible number of bytes to transmit over the wire. Your range 30-1200 requires two bytes, as a single byte can only represent numbers from -128 to +127 or 0 to 255. It doesn't quite fill out 2 bytes - you only need 11 bits, for numbers as large as 2047. That would suggest you could do it all in 33 bits, which is just one bit bigger than 4 bytes. But let's keep it simple and make it 3 two-byte integers.

I see you're using floats. A standard float requires 4 bytes. What precision do you need? Maybe you can use a fixed-point system, so to represent numbers as large as 6553.6 you multiply the number by 10 before transmitting it as an integer and then divide by 10 at the receiving end.

So I estimate you need 6 bytes as your data payload. There must be some kind of "frame" or "start marker" to let the receiver know where the values start and stop. Usually you choose a byte that cannot exist in the data. Easy with text, not so easy with binary data. You said the range is 30-1200 so maybe zero can never appear in the real data? That makes it easy. Just send a 2-byte zero, followed by the XYZ data. So the entire packet is 8 bytes long.

At 115200 baud, it takes 694 microseconds to send a single packet. (Remember a Serial byte requires a start bit and a stop, so it takes 10 bit periods to send one byte of data.) Or 1440 packets/sec.

1000RPM is relatively fast for a stepper, but not unreasonable. Assuming 200 steps per revolution, that's 3333 steps per second. Hmm... more than twice as fast as the Serial can send the data.

Even if you "pack" the data down to 11 bits, you're not going to get where you want to be using 115200 baud.

If you can get 500000 baud working then you have a chance of sending this quantity of data in the time required.

10

Hi, wow thanks.. I'm sort of an intermediate level arduino user and unfortunately never really learned how the memory and stuff works with the arduino but this problem is making me learn a great deal.
and my float value to 1 or 2 decimal points is good, I'll also try the 500000 baud and hopefully it works

11

kikisarpong:
and my float value to 1 or 2 decimal points is good,

Pick one. Don't give us a range. But if you pick "2" that pushes you out of 16-bit ints into something bigger and you might not be able to get there from here.

12

Just to double check and make sure you actually need this insane amount of data:

Why do you need data reported to the processing sketch at 3K packets/sec? What does the processing sketch use the data for - do you really need to know the sensor data at every stepper motor position?

If you are only "drawing" a graph or depicting the data on the screen for user debugging/status reporting, even 100 packets/sec would be fine. Now, if you're trying to do some sort of frequency analysis or digital signal processing of the sensor data (i.e. FFT) - then you may actually have a need for such a high data rate - otherwise don't worry about it.

13

I'm mapping an environment, and reproducing a digital map/ dense point cloud of the said environment on processing, and as my stepper motor rotates , I need the lidar data at every step to get accurate information of where objects are located, that is why I need the data at every step for 800 steps
P.s I changed the float to INT.
and changing the data to byte using "power_broker" (sendData) function sped up my program to a significant speed (haven't measured yet) but still not fast enough, will keep doing some tests to see what I can do to increase the speed more.
Thanks

14

kikisarpong:
that is why I need the data at every step for 800 steps

Give an example of the message that has to be sent for every step.

How many steps per second do you want to take?

...R

15

What lidar?

16

The lidar type is the tfmini lidar
Im hoping to get about 1200rpm i.e 20 steps per sec(hoping)
and an example of the message sent is

void sendData()
{
  byte buf[] = {'<', 0, 0, 0, 0, 0, 0, '>'};
  byte len = 8;

  buff[1] = (x_val >> 8) & 0xFF;  //MSB
  buff[2] = (x_val) & 0xFF;       //LSB

  buff[3] = (y_val >> 8) & 0xFF;  //MSB
  buff[4] = (y_val) & 0xFF;       //LSB

  buff[5] = (z_val >> 8) & 0xFF;  //MSB
  buff[6] = (z_val) & 0xFF;       //LSB

  Serial.write(buf, len);
  
  return;
}
17

hmm, maybe slow the step rate?

What about sending two pieces of data and only one marker (i.e. byte buf[] = {'<', 0, 0, 0}). You could send the scan index and the distance measured. The step index can be correlated to a elevation and rotation angle pair.

For instance:

Let's say I have two steppers - 1 for rotation angle "ρ" and 1 for elevation angle "θ". Because the steps of both motors are repeatable and predetermined during the coding process, we can directly correlate a step index with a given (ρ, θ) pair.

For our example let's assume the following: We will have 4 steps per point cloud. Let's also say that the steps for the point cloud is the following:

Point 1 --> (ρ = 0°, θ = 0°)
Point 2 --> (ρ = 45°, θ = 45°)
Point 3 --> (ρ = 90°, θ = 90°)
Point 4 --> (ρ = 120°, θ = 120°)

Obviously this "point cloud" will not give you a good idea of the environment around you, but this is just an academic exercise to explain my original idea of only needing to send 2 pieces of data. Notice that if I send the following packets, I can reliably construct a point cloud:

packet 1 --> {'<', 1, 0, 1}
packet 2 --> {'<', 2, 0, 9}
packet 3 --> {'<', 3, 0, 7}
packet 4 --> {'<', 4, 0, 1}

The following point cloud would be (r is distance):

Point 1 --> (ρ = 0°, θ = 0°, r = 1)
Point 2 --> (ρ = 45°, θ = 45°, r = 9)
Point 3 --> (ρ = 90°, θ = 90°, r = 7)
Point 4 --> (ρ = 120°, θ = 120°, r = 1)

Does this make sense? You just need to know beforehand what angles you are measuring the distances at - and their order.

18

oooh..took me a minute but I think I get what you mean. My thought process initially was the more information you get at every step, the higher the accuracy of the point cloud.
and I'm using a stepper for rotation and a servo for elevation data because a servo uses less power and I wanted to reduce my power requirement.
but i'll test out what youre saying and see the results, thanks

19

kikisarpong:
The lidar type is the tfmini lidar
Im hoping to get about 1200rpm i.e 20 steps per sec(hoping)
and an example of the message sent is

void sendData()

{
  byte buf[] = {'<', 0, 0, 0, 0, 0, 0, '>'};
  byte len = 8;

buff[1] = (x_val >> 8) & 0xFF;  //MSB
  buff[2] = (x_val) & 0xFF;      //LSB

buff[3] = (y_val >> 8) & 0xFF;  //MSB
  buff[4] = (y_val) & 0xFF;      //LSB

buff[5] = (z_val >> 8) & 0xFF;  //MSB
  buff[6] = (z_val) & 0xFF;      //LSB

Serial.write(buf, len);
 
  return;
}

Your message is 8 bytes long. 20 of those per second needs only 1600 baud. 9600 baud would cover that comfortably. That's slower than a crawl for an Arduino.

I'm confused when you say "steps" and "RPM". Most stepper motors have 200 steps per revolution, so your "steps" is wrong by a factor of 200.

20

sorry I meant 20 revolutions per sec and above 1200 revolutions per minute