ARDUINO YUN REV 2 - Speed writing is slow (Bridge)

Hello,

In my project, I use an Arduino Yun Rev 2 and an accelerometer (I2C), and need the WIFI to download files created by the Yun.

Accelerometer used :

Here my questions :

I want to write values in SD at 100~400 datas per sec, but the Bridge is too slow for that. To avoid the FileIO, I think I can use SD module by connecting in SPI
https://forum.arduino.cc/index.php?topic=492507.0

• It's really easier to use another module SD card ?
• If it is, I already know the SPI connectors are in ICSP,

I've gotten my best performance by doing as much processing as possible on the Linux side. The Yun is a different beast than other Arduinos, and using it to its best potential requires a different way of thinking.

But here, it seems the performance are better using Linux side, so how can we use this to write files ? I thought the I2C is connected to the 32U4, so it need to pass though the bridge no ?
(I start with Linux this week, so there are a lot of thing I can't manage without a complete guide)

Thank you if you can help

PS : I'm not good in english writing (but I understand well :p)

The Yun's SD card is attached to, and managed by, the Linux system. The I2C pins on the header connector are only connected to the AVR processor, and can only be controlled by a sketch. Something needs to get the data from the sketch to the SD card.

When you use FileIO to write to a file, you are using the Bridge library to send data over the serial port that is between the two systems. This data includes information about the file to be opened, as well as the data for the file. As you have found, it is a rather inefficient way to write to the SD card.

File operations performed directly by code on the Linux processor are much faster. But you still need to get the data from the sketch to the Linux processor so that code can write the data. The way I normally do that is through the Process class.

I write a Python script (you can use any language you prefer) that reads data from standard input, processes and formats it as necessary, and writes it to the SD card. The sketch uses the Process class to start the Python script, and then in the loop() function it reads the external data and writes it to the process object. The Python script receives that data, and writes it to the file.

Since the Linux processor is so much faster and more powerful than the sketch processor, I do the minimum amount of processing in the sketch, and try to do as much as possible on the Linux side. But I temper that with the amount of data, and do only as much as processing on the sketch side as necessary to minimize the amount of data that goes over the bridge. For example, in one of my projects, I want to get an average reading of a quickly varying waveform. So instead of sending a lot of data down to the Linux side, my sketch quickly reads and totals 1000 raw analog samples, and then sends that one total value down to the Linux side. The Linux side then divides by the number of samples, scales the raw reading to the proper engineering units and then stores the data. In this way, there is no need to send all of the samples down to the Linux side, but also the sketch doesn't have to do the floating point math to scale the data, and it certainly doesn't have to scale each individual sample.

So it can be a delicate balance of how much processing you want to do on each side. The I2C data collection must happen in the sketch. The file writing operations should happen on the Linux side. To determine where the rest of the processing should be, you need to analyze your data and the required processing and identify the minimum amount of processing needed to generate the minimum amount of data. That processing should happen in the sketch, the data gets passed to the Linux side using the process object, and the rest of the processing and the file writing happens on the Linux side.

If there is data going the other way as well, that can also be handled by the same process connection: anything that the Linux side code writes to standard output can be read by the sketch and used to control outputs. In the example I mentioned above, I am controlling some low voltage AC lighting. A current transformer is used to measure the lighting current, and the values are read and averaged by the sketch and sent to the Linux side, while the Linux side sends on/off commands up to the sketch which is used to control a power relay to control the lights. Most of the processing and all of the control decisions happen on the Linux side, while the sketch is a slightly intelligent I/O processor.

Using the Process class to communicate with the Linux side and doing the file access in Linux should significantly speed things up. Minimizing the amount of data sent to the Linux side will help. While the Process communications are more efficient than FileIO, you may still have trouble with the high data rates that you desire.

Thank you for your time

At first, I have collect samples in arduino sketch (300~400 samples are maximum, because memory are really limited) and write in SD card with FileIO, but it is equivalent of 3-4sec of measure and 10sec of writing, and it isnt enough (I think I can deal with it, but more is better).
Here, it process a lot in arduino sketch

But I don't really get it, for one of your project, when you read and total 1000 samples, then Linux divided by the number of samples, it seem each value is averaged ?

BTW, do you have a guide for beginner like me, to run python in Arduino Yun ?

hvlk:
But I don't really get it, for one of your project, when you read and total 1000 samples, then Linux divided by the number of samples, it seem each value is averaged ?

Yes, I'm sorry if I wasn't clear, but my intent on that project was to quickly gather and average 1000 samples. I could've collected 1000 samples, scaled each of those to floating point volts, and sent them all to the Linux side. But that's a lot of data being transferred. So, to save data, I could've averaged those scaled values by adding them all together and dividing by the number of samples, but that's still a lot of computation going on in the sketch. You get the same result by adding together the raw samples, dividing by the count, and then scaling the single value. That way, only a single value is sent. But why do the division and the final scaling in the sketch? To keep the sketch as simple as possible, I just send the total, and let the Linux side divide and scale it.

That probably isn't directly useful in your application. But I present the idea as a use case of analyzing the data, and looking for the most efficient way of transporting the data down to the Linux side. You should do a similar analysis of your data requirements, and look for the most efficient way to move the data. The simple idea is to simply capture everything to a data file, and figure it out later. But that can be a lot of data, and some simple pre-processing might be able to help reduce the load.

BTW, do you have a guide for beginner like me, to run python in Arduino Yun ?

Python is built into the Yun. To get started, all you need to do start an SSH session with the Yun, type "python" at the command line, and you will be in the interactive Python environment. You can then follow along with most any learning Python tutorial.

To run a script in a file, at the command line just type "python file.py" where file.py is the name of the file containing your Python script.

So,

Due to the amount of data I need to collect, I decided to use a SD module as a temporary memory. So I connected (by SPI) a SD module and included <FileIO.h> and <SD.h> in my Arduino sketch.

It seems to work properly, I can create files of 10min.

To transport datas through the bridge, I read the file line by line and send it to Linux side.

But, I still don't know if this method is viable...

hvlk:
But, I still don't know if this method is viable...

Well, given that the definition of viable is "capable of being done with means at hand and circumstances as they are" and given that it is working for you and meeting your needs, I'd say that the method is indeed viable. 8)

Good work!