Interface Arduino-Raspberry Pi 3 for data logger project

powergravity · January 26, 2017, 8:55am

Hello,

I am currently working on a motorcycle data logger project based on Arduino hardware. Actually I use a SD connected to the Arduino to store the data. But I am thinking to send them to the Pi for different reasons:

store data is faster.
Pi is more performant and allows more freedom to do computation.
...

I read several topics about PI/arduino interfacing and it seems not so easy in my case.

To summarize my data logger's code, it reads the signals from a steering sensor, two wheel speed sensors and an IMU each 10 ms (100Hz) and from the GPS each 50 ms (20Hz).

According to the flow rate, I am looking for a fast way to communicate between the two devices. I saw that SPI and I2C are faster than Serial but it's quite impossible to set the PI as a slave.

So I would appreciate your feedback.

Thank you in advance,
Pm

MrMark · January 26, 2017, 5:17pm

powergravity:
To summarize my data logger's code, it reads the signals from a steering sensor, two wheel speed sensors and an IMU each 10 ms (100Hz) and from the GPS each 50 ms (20Hz).

What is this in bytes per second? It's not obvious to me that serial over USB can't support this and it's the most straightforward approach.

GypsumFantastic · January 26, 2017, 7:27pm

powergravity:
According to the flow rate, I am looking for a fast way to communicate between the two devices. I saw that SPI and I2C are faster than Serial but it's quite impossible to set the PI as a slave.

The Pi wouldn't have to be Slave. It could take charge of the bus and demand data from a slave Arduino, the same way that the Arduino as master collects data from an SPI-interfaced device like a sensor etc.

But I agree with MrMark. Serial over USB is probably the best way to go if it's fast enough for what you need.

Robin2 · January 26, 2017, 7:45pm

This Python - Arduino demo may help get you started. As written it uses 9600 baud but an Arduino can communicate at 500,000 baud. I don't have an RPi myself.

Throughput on the USB system may be better with data chunks of 64 bytes or more. So it might be worth experimenting with storing a few measurements on the Arduino and then sending them together. If you don't send at least 64 bytes I think the USB waits up to 1 millisec before sending.

...R

powergravity · January 26, 2017, 7:48pm

Thank you for your answers.

What is this in bytes per second?

Around 13 kB/s. Do you think that USB is fast enough?

The Pi wouldn't have to be Slave. It could take charge of the bus and demand data from a slave Arduino, the same way that the Arduino as master collects data from an SPI-interfaced device like a sensor etc.

Very interesting. I guess with this method the Pi has to periodically take the Arduino data ?

Thank you,
Pm

powergravity · January 26, 2017, 7:55pm

Throughput on the USB system may be better with data chunks of 64 bytes or more. So it might be worth experimenting with storing a few measurements on the Arduino and then sending them together. If you don't send at least 64 bytes I think the USB waits up to 1 millisec before sending.

According to your comment I think a good way to proceed would be to create a buffer able to store 50 ms of sensor readings:
5x10 ms IMU
5x 10 ms Wheel speed sensor
5x 10ms Steering sensor
1x50ms GPS
and then send the data to the Pi.

Thank you,
Pm

CrossRoads · January 26, 2017, 8:24pm

Will all that fit in the 2K SRAM of a '328P, or would the 16K SRAM of a '1284P be a better fit?

"Around 13 kB/s. Do you think that USB is fast enough?" USB 2.0 is like 4 megabits/second. 16 MHz Arduino can do serial at 115200, and 250,000 bits/second (but not 250K to the serial monitor in the IDE last I checked). So Serial to the USB/Serial chip would seem to be fast enough.

MrMark · January 26, 2017, 8:31pm

powergravity:
Thank you for your answers.
Around 13 kB/s. Do you think that USB is fast enough?

The Raspberry Pi USB is easily fast enough. From personal experience it will sustain several megabytes per second including meaningful processing of that received data.

The bottleneck is going to be on the Arduino. The Arduino serial.print() isn't particularly fast, but it should be fast enough for your application.

Edited to add: My assumption is that you are using an Arduino Uno or similar where the path is TTL serial out of the Atmega328 to the on-card serial to USB converter to the Raspberry Pi.

powergravity · January 27, 2017, 9:45am

Hi,

Thank you for your answers. This forum is definitively a great community.

I did some tests over USB serial communication, it is seems fast enough from Arduino side with NeoSerial library. I guess according to the faster Pi's clock it will not be a problem to read the data.

I need advices to properly design the communication between the two devices.
As a reminder, the Arduino reads IMU, Wheel speed sensor and Steering sensor every 10 ms and GPS every 50 ms.

I am not able to choose the best way to do between to send the sensor's readings as soon as they are available or to store some readings and then to send a larger amount of data to the pi.
Any suggestion?

The objective is to be able to log the data and to do some computations with the raw data.

Please find attached my Arduino sketch which actually logs the data on the SD with some missing readings (This is one of the reasons why I am trying to log them on the Pi).
It simulates the whole of the sensors ( for convenience data are fixed ). It can give you an idea about what I am trying to send on the Pi.

Thank you,
Pm

DataLogger.zip (158 KB)

powergravity · January 29, 2017, 6:24am

Hello,

The bottleneck is going to be on the Arduino. The Arduino serial.print() isn't particularly fast, but it should be fast enough for your application.

I did some tests over USB serial communication, it is seems fast enough from Arduino side with NeoSerial library.

You were right MrMark. I thoroughly analyzed this point and serial writings take took much time and inevitably lead to missing sensor readings.
Regarding the serial writing time, you can see that write the data is faster during the 10 first seconds. I am not able to understand why :-\

I think I should change the way to communicate between the Arduino and the Pi.
Let's go to try SPI, if you have some references with similar idea, I would be grateful if you could mentioned it.

Thank you very much,
Pm

Robin2 · January 29, 2017, 9:33am

powergravity:
I think I should change the way to communicate between the Arduino and the Pi.

Before spending time on that please post an example of a message that you are presently sending.

It may be possible to shorten the message without losing any information.

And what baud rate are you using?

...R

powergravity · January 29, 2017, 10:00am

an example of a message that you are presently sending

I am trying to send this kind of message:

W,12196416,0,0
S,12197328,295
L,12206836,9652,1,89
W,12206416,0,0
S,12206524,295
I,12186756,70,20,104,87,15,16,5759,9,25,17,41,16012,12186756
I,12196756,70,20,104,87,15,16,5759,9,25,17,41,15956,12196756
W,12216420,0,0
L,12217104,9792,0,0
S,12218088,295
I,12206756,70,20,104,87,15,16,5759,9,25,17,41,14008,12206756
W,12226436,0,0
L,12227124,4860,0,0
S,12228108,295
I,12216756,70,20,104,87,15,16,5759,9,25,17,41,14032,12216756
G,12236408,20:59:31.95,48.6082867,2.41341170,0,2050
L,12238560,6356,0,0
W,12240356,0,0
S,12240368,295
I,12226756,70,20,104,87,15,16,5759,9,25,17,41,14960,12226756
W,12246440,0,0
S,12247144,295
I,12236756,70,20,104,87,15,16,5759,9,25,17,41,14008,12236756
L,12253360,9552,0,0
W,12256416,0,0
S,12257116,295
I,12246756,70,20,104,87,15,16,5759,9,25,17,41,14024,12246756
L,12263368,4852,0,0
W,12266424,0,0
....

The first letter for the sensor's name( I for IMU, W for Wheel speed sensor, S for Steering sensor, L for debugging) then it follows the reading time and the sensor's data.

I set the baud rate at 115200.

Thank you,
Pm

Robin2 · January 29, 2017, 1:22pm

I have just picked this example because it is long

I,12186756,70,20,104,87,15,16,5759,9,25,17,41,16012,12186756

Can you explain what each number represents.

It would be possible with a slightly more complex program at each end to send the data in binary format. Then those 60 bytes could reduce to 23 bytes. But sending binary data makes debugging significantly harder so I try to avoid it.

Try a much higher baud rate. 500,000 baud works nicely on an Arduino.

...R
PS. For one of my projects I convert numbers into 2, 3 or 4 digit values to base 64. 64³ can represent a little over 260,000 - 3 chars instead of 6. By adding 48 to the digits I can produce printable characters. And converting one of those 3 digit values back into a LONG is about 10 times faster than the atoi() function.

MrMark · January 29, 2017, 4:47pm

powergravity:
Around 13 kB/s. Do you think that USB is fast enough?

I thoroughly analyzed this point and serial writings take took much time and inevitably lead to missing sensor readings.

Serial asynchronous data transfer requires at least 11 bits per byte, so you need a baud rate setting greater than (13k * 11) = 143000 at a minimum. Thus your serial.begin(??) needs to be 230400 or greater.

powergravity · January 29, 2017, 5:37pm

Thank you for your support.

Can you explain what each number represents.

The function to write the data is the following:

void writeData() {
  
  bool     wroteSomething   = false;  // This really should be local!
  uint32_t startWritingTime = micros();

  while (WSSreadings.available()) {
    // Write one 'W' record with both front and rear counts
    WSSreading_t one = WSSreadings.read();
    
    DEBUG_PORT.print( F("W,") );
    DEBUG_PORT.print( one.t );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.front );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.println( one.rear );
    wroteSomething = true;
  }

  while (SteeringReadings.available()) {
    // Write one 'S' record
    SteeringReading_t one = SteeringReadings.read();
    
    DEBUG_PORT.print( F("S,") );
    DEBUG_PORT.print( one.t );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.println( one.ADCvalue );
    wroteSomething = true;
  }

  while (IMUreadings.available()) {
    // Write 'I' record
    IMUreading_t one = IMUreadings.read();

    DEBUG_PORT.print( F("I,") );
    DEBUG_PORT.print( one.t );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.rollRate );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.pitchRate );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.yawRate );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.roll );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.pitch );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.yaw );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.aX );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.aY );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.aZ );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.print( one.dt2 );
    DEBUG_PORT.print( ',' );
    DEBUG_PORT.println( one.dt3 );
    wroteSomething = true;
  }

  while (gps.available()) {
    // Write one 'G' record
    gps_fix one = gps.read();
  
    DEBUG_PORT.print( F("G,") );

    if (one.valid.time) { 
         
      // When the whole second rolls over, use the previously-stored PPS timestamp
      if (one.dateTime_cs == 0) {

        cli();
          if (nextPPStime) { // PPS was detected
            lastPPStime = nextPPStime;
            nextPPStime = 0; // reset so we know if a PPS was detected for the next second
          } else { // no PPS detected...
            lastPPStime += 1000000UL; // ...just add one second to the last one
          }
        sei();

        DEBUG_PORT.print( lastPPStime );
        
      } else
        // Calculate an offset from the previous PPS
        DEBUG_PORT.print( lastPPStime + one.dateTime_cs * 10000UL );

      DEBUG_PORT.print( ',' );
    
      if (one.dateTime.hours+1 < 10)     // when hours==9, don't print 010
          DEBUG_PORT.print( '0' );
      DEBUG_PORT.print( (one.dateTime.hours+1) % 24 ); // This is not UTC, it's local time.
      DEBUG_PORT.print( ':' );
      if (one.dateTime.minutes < 10)
          DEBUG_PORT.print( '0' );
      DEBUG_PORT.print(one.dateTime.minutes);
      DEBUG_PORT.print( ':' );
      if (one.dateTime.seconds < 10)
          DEBUG_PORT.print( '0' );
      DEBUG_PORT.print(one.dateTime.seconds);
      DEBUG_PORT.print( '.' );
      if (one.dateTime_cs < 10)
         DEBUG_PORT.print( '0' ); // leading zero for .05, for example
      DEBUG_PORT.print(one.dateTime_cs);
    }
    DEBUG_PORT.print( ',' );
    if (one.valid.location)
      printL( DEBUG_PORT, one.latitudeL() );
    DEBUG_PORT.print( ',' );
    if (one.valid.location)
      printL( DEBUG_PORT, one.longitudeL() );

    DEBUG_PORT.print( ',' );
    if (one.valid.speed)
      DEBUG_PORT.print( one.speed_mkn() );
    DEBUG_PORT.print( ',' );
    if (one.valid.heading)
      DEBUG_PORT.print( one.heading_cd() );
    DEBUG_PORT.println();
    wroteSomething = true;
  }

  if (wroteSomething) {
    // Accumulate how long write data took
    currentTime  = micros();
    writingTime += currentTime - startWritingTime;

    // Occasionally write an 'L record
    if (currentTime - lastLrecord > LRECORD_PERIOD) {
      lastLrecord = currentTime; // It's ok if these times creep a little

      DEBUG_PORT.print( F("L,") );
      DEBUG_PORT.print( currentTime );
      DEBUG_PORT.print( ',' );
      DEBUG_PORT.println( writingTime );

      currentTime  = micros();
      writingTime  = currentTime - lastLrecord; // this time gets added to *next* logging times
    }
  }
    
} // writeData

Before being written the data are stored in a ring buffer.
I set the baud rate to 250000 and the results are better but I still miss some readings.

Do you know another terminal I could install on my mac to work with higher baud rate than 250000?

Thank you,
Pm

DataLoggerMoto_Rasp_v2.ino (21 KB)

Robin2 · January 29, 2017, 9:17pm

powergravity:
The function to write the data is the following:

That is not the answer I wanted.

I would like to know (in English not code) what each number represents - for example you grandmother's height and your daughter's birth date.

...R

powergravity · January 30, 2017, 6:11am

Hi,

That is not the answer I wanted.

Sorry let me explain.
First I did a mistake, I took the wrong txt file, the recordings look like:

S,4590548,389
L,4591772,4280
I,4590184,0,0,-2,-16,-13,0,0,0,-11,2380,4590328
G,4547248,18:07:52.85,48.6084317,2.4133433,0,0
W,4600372,0,0
S,4600392,402
L,4602560,7332
I,4600184,0,1,0,-16,-13,0,0,0,-8,3196,4600324
G,4597248,18:07:52.90,48.6084300,2.4133433,0,0
W,4611460,0,0
S,4611472,393
L,4613244,6456
I,4610184,0,2,1,-16,-13,0,0,0,-8,7720,4610272
W,4619880,0,0
S,4620540,399
I,4620184,0,2,0,-16,-13,0,0,1,-9,1496,4620300
L,4623680,5900
W,4629852,0,0
S,4630420,386
.....

For a 'G' recording:
G,4547248,18:07:52.85,48.6084317,2.4133433,0,0
GrandMother, DaySinceBirth, BedTime, Height, Weight, NumberOfRemainingTeeth, ...

More seriously,

For an IMU recording:
I, readingTime, rollRate, pitchRate, YawRate, roll, pitch, yaw, aX, aY, aZ, timeToRead, timerForDebugging

For a wheel speed sensor recording:
W, readingTime, frontCounter, rearCounter

For a stearing sensor recording:
S, readingTime, AnalogReading

For a debugging recording:
L, readingTime, timeToWriteOnSerial

For a GPS recording:
G, readingTime, UTC, longitude, latitude, speed, heading

Thank you very much,
Pm

Robin2 · January 30, 2017, 10:02am

powergravity:
For an IMU recording:
I, readingTime, rollRate, pitchRate, YawRate, roll, pitch, yaw, aX, aY, aZ, timeToRead, timerForDebugging

OK. We are making some progress.

Now what is the datatype and range of values that must be allowed for for each value? For example, maybe the roll is always an int between 0 and 360.

In this particular example what is the difference between readingTime, timeToRead and timeForDebugging? Do you really need all three?

I can't remember if you have told us how many messages you need to send every second?

...R

powergravity · January 30, 2017, 1:52pm

Robin2:
OK. We are making some progress.

Now what is the datatype and range of values that must be allowed for for each value? For example, maybe the roll is always an int between 0 and 360.

I acquire raw data from the BNO055 IMU by using this function without the floating conversion. I process the data off line with Matlab.

  readLen((adafruit_bno055_reg_t)vector_type, buffer, 6);

  x = ((int16_t)buffer[0]) | (((int16_t)buffer[1]) << 8);
  y = ((int16_t)buffer[2]) | (((int16_t)buffer[3]) << 8);
  z = ((int16_t)buffer[4]) | (((int16_t)buffer[5]) << 8);

  /* Convert the value to an appropriate range (section 3.6.4) */
  /* and assign the value to the Vector type */
  switch(vector_type)
  {
    case VECTOR_MAGNETOMETER:
      /* 1uT = 16 LSB */
      xyz[0] = ((double)x)/16.0;
      xyz[1] = ((double)y)/16.0;
      xyz[2] = ((double)z)/16.0;
      break;
    case VECTOR_GYROSCOPE:
      /* 1rps = 900 LSB */
      xyz[0] = ((double)x)/900.0;
      xyz[1] = ((double)y)/900.0;
      xyz[2] = ((double)z)/900.0;
      break;
    case VECTOR_EULER:
      /* 1 degree = 16 LSB */
      xyz[0] = ((double)x)/16.0;
      xyz[1] = ((double)y)/16.0;
      xyz[2] = ((double)z)/16.0;
      break;
    case VECTOR_ACCELEROMETER:
    case VECTOR_LINEARACCEL:
    case VECTOR_GRAVITY:
      /* 1m/s^2 = 100 LSB */
      xyz[0] = ((double)x)/100.0;
      xyz[1] = ((double)y)/100.0;
      xyz[2] = ((double)z)/100.0;
      break;
  }

  return xyz;
}

So IMU data are int16_t and their ranges are the following:
-2880<roll<2880 (-180/180 deg)
-1440<pitch<1440 (-90/90 deg)
0<yaw<5760 (0/360 deg)

In this particular example what is the difference between readingTime, timeToRead and timeForDebugging? Do you really need all three?

You are right I don't need the three. ReadingTime is essential because it is the time when the date are read and I need it to sort the recordings.

But do you think that delete 2 data from IMU message could significantly increase the speed? I am sceptic....

I can't remember if you have told us how many messages you need to send every second?

I need to send around 13 kB each second.

Thank you,
Pm

Robin2 · January 30, 2017, 2:06pm

powergravity:
So IMU data are int16_t and their ranges are the following:
-2880<roll<2880 (-180/180 deg)
-1440<pitch<1440 (-90/90 deg)
0<yaw<5760 (0/360 deg)

Am I correct then to think that this
"I, readingTime, rollRate, pitchRate, YawRate, roll, pitch, yaw, aX, aY, aZ, timeToRead, timerForDebugging"
might be as follows in the worst case (longest message)
"I,ttt,-2880,-1440,5760,-2880,-2880,-1440,5760,xxx,yyy,zzz"
As I write that I see that you have not explained the ranges for the things I have called ttt, xxx, yyy and zzz

You are right I don't need the three. ReadingTime is essential because it is the time when the date are read and I need it to sort the recordings.

But do you think that delete 2 data from IMU message could significantly increase the speed? I am sceptic....

I thought the whole purpose of this Thread was because you are having a problem getting all the data delivered. Every improvement should be grabbed with both hands. And it would probably only take you 10 or 15 minutes to set up a test to see if there is an improvement.

I need to send around 13 kB each second.

Again, that does not answer my question which was how many messages per second

...R

Topic		Replies	Views
Hi Speed serial data transfer PC to Arduino via USB2 Interfacing w/ Software on the Computer	43	773	October 31, 2025
Trouble sending large amounts of data over serial Programming	34	626	June 3, 2024
Raspberry PI and several Arduinos General Guidance	38	2004	May 24, 2024
Logic Daten an serielle Schnittstelle senden Deutsch	53	564	November 24, 2025
Control loop Issues Python Arduino communication General Guidance	10	123	October 24, 2025

Interface Arduino-Raspberry Pi 3 for data logger project

Related topics