Yun SD read/write is slow

I am trying to use the onboard SD card to log data from some sensors. The problem I am running into is that I need to log at 200Hz or faster. My best efforts so far on the Yun are only getting about 40Hz. I have done some searches and it appears that the bottleneck is the bridge library. Someone mentioned that they fixed the problem by bypassing bridge but don’t mention ho they did that.

My proposed solution is to send data from the MCU over serial1 at 115200 baud and run a python script to read the data and write it into the SD card. So far I have been able to send data over serial and read it in python and I am also able to write to the SD card from python. However, I have not been able to put all the pieces together successfully yet. I am pretty new to python so I am fumbling around quite a bit. I have spent a couple of days on this and before I go further I am interested to get suggestions on if I am going at this the right way?

fb4: I am trying to use the onboard SD card to log data from some sensors. The problem I am running into is that I need to log at 200Hz or faster. My best efforts so far on the Yun are only getting about 40Hz. I have done some searches and it appears that the bottleneck is the bridge library. Someone mentioned that they fixed the problem by bypassing bridge but don't mention ho they did that.

My proposed solution is to send data from the MCU over serial1 at 115200 baud and run a python script to read the data and write it into the SD card. So far I have been able to send data over serial and read it in python and I am also able to write to the SD card from python. However, I have not been able to put all the pieces together successfully yet. I am pretty new to python so I am fumbling around quite a bit. I have spent a couple of days on this and before I go further I am interested to get suggestions on if I am going at this the right way?

Which Yun are you using. An arduino.cc or arduino.org Yun? Which Linux. OpenWRT or Linino? Which version of OpenWRT or Linino?

It is difficult to answer your question without the above information.'

Your proposed solution does sound fast and I for one would certainly be interested in seeing some code from the finished setup if that method comes to fruition. Hopefully some other people will be able to help you on that.

If you have not already done something similar, I’d like to note that you can get somewhat faster speeds using FileIO.h by storing an array of data and then writing it all at once.

if (dataFile) {
    dataFile.println(data);
    dataFile.close();
  }

vs.

if (dataFile) {
    for(int i=0;i<500;i++) dataFile.println(dataArray[i]);
    dataFile.close();
  }

Whether that is feasible for you obviously depends on how large each data set is, since we’ve got limited RAM to work with. If you can do it, however, it spreads the overhead of making the file connection out over a larger number of samples.
In a very basic speed test:
Writing “123456” 500x sequentially: 19389ms
Writing “123456” 500x from array without closing file: 10213ms

Still not close to 200Hz though.

artisticForge- I looked at my Yun and it says .org on it. What is the difference with the .cc version?

I am brand new to Linux and am trying to learn so I am not sure which version I am running. I installed the Serial output on the MCU side and rebooted it and in the boot sequence I see up top that it says "U-Boot 1.1.5-linino-gd3609a72-dirty (Sep 11 2015 - 07:18:37)" but later in the boot sequence I see "Image Name: MIPS OpenWrt Linux-3.3.8" . So I am a little confused and hope you can help me identify what I have.

DarkSabre- I thought of that and I don't have enough memory to collect all the samples and then send them. I need at least 2000 samples which chews up to much memory. Good thought though!

Just an update on the project. I was able to get this working and bypass the bridge library. On the Arduino side I just write to Serial1 and then on the Linux side I read the serial and write it to the SD card. The problem I am having is that it is not any faster (at least not reliably). I need to keep a delay of 30 milliseconds between writes which equates to about 33Hz and I am looking for 200Hz or better. I think the issue is the baud rate which is 9600. However, every time I try to use a higher baud rate I get serial errors. I posted my code below as well as the error I get when I try to run a higher baud rate. I feel like I am so close and would appreciate any suggestions on how to run a higher baud rate.

Python:

ser = serial.Serial('/dev/ttyATH0', 9600)
file = open("/mnt/sd/logs/sdHello.txt", "w")

i = 0

while i == 0:
	inbyte = ser.read()
	data = str(inbyte)
	file.write(data)
	if data == '!':
		i = 1 #gets us out of the loop

file.close()

print("shut down")
ser.write("shut Down\r")

Arduino code:

void setup() {
  Serial1.begin(9600);
  SerialUSB.begin(9600);
  delay(1000);
  
}

void loop(){
  if (Serial.available() > 0) {
    Serial.println("writing");
    char inByte = Serial.read();
    for(int i = 0; i < 10; i++){
      Serial1.println('f');
      delay(30);
    }
    delay(100);
    Serial1.println('!');
    delay(5000);
  }

  if(Serial1.available() > 0){
    char inbyte = Serial1.read();
    Serial.print(inbyte);
  }
  
}

Error I get when I try to run higher baud rate

root@Arduino:~# python sdWrite.py
Traceback (most recent call last):
  File "sdWrite.py", line 14, in <module>
    x = ser.read()
  File "/usr/lib/python2.7/site-packages/serial/serialposix.py", line 475, in re           ad
    raise SerialException('device reports readiness to read but returned no data            (device disconnected or multiple access on port?)')
serial.serialutil.SerialException: device reports readiness to read but returned            no data (device disconnected or multiple access on port?)
root@Arduino:~# python sdWrite.py
Traceback (most recent call last):
  File "sdWrite.py", line 14, in <module>

Well this seems pretty cool; I hadn’t tried the direct communication before.

I’m not a python guy, but I did this test with PHP.

byte i = 0;
int j = 0;
unsigned long timer;

void setup() {
  Serial1.begin(9600);
  delay(5000);
  timer = millis();
}

void loop() {
  while (millis() - timer < 60000) {
    Serial1.println(i);    
    if (i == 255) {
      i = 0;
      j++;
      Serial1.println(j);
    }
    i++;
  }
}
<?php
/*packages required
php5
php5-cli
php5-mod-dio
*/

$fd = dio_open('/dev/ttyATH0', O_RDWR | O_NOCTTY | O_NONBLOCK);

dio_fcntl($fd, F_SETFL, O_SYNC);

dio_tcsetattr($fd, array(
  'baud' => 9600,
  'bits' => 8,
  'stop'  => 1,
  'parity' => 0
)); 

echo "booted";

while (1) {

  $data = dio_read($fd, 256);

  if ($data) {
      echo $data;
  }
}
?>

The PHP console printed out 12594 bytes in the 60 seconds, so about 210 samples per second.

@ 19200 baud on both programs it runs 25151 bytes in the 60 seconds, so about 419 samples per second.
@ 38400 baud on both programs it runs 50251 bytes in the 60 seconds, so about 837 samples per second.

Notably I’m not doing any file writes.
The PHP output doesn’t skip any bytes, but does trim off some zeros. Not sure about that part.

I’m honestly not sure about your Python errors, but I’m eager to try this on one of my projects now…

I did some additional testing and found that the delay in my setup was the time it took to write to the SD card. I also found that I could not run any baud rate other than 9600. After some research I found that I had to shut off the bridge connection on the Linux side. Once I did this I was able to run 115200 baud without an issue. To fix the lag in writing to the SD card I wrote a python script which takes the values in over serial1 and put them into a list. After each value I send a \n as a delimiter so python knows where to break the values for the list. Then I send a terminating character to let python know that the transmission is over and to write the values. Then I just read the list one by one and write into the SD card. It seems to be very reliable and without trying to optimize it I am getting over 400Hz transmission rate. The one thing I did notice is that if i use Serial1.println the time to transmit is nearly double. As a result I am using Serial1.print and then just send a delimiter character which is much faster.