I'm trying to see whether Arduino (with appropriate peripherals/shields) is a suitable candidate for a project that requires logging/recording ethernet or serial data to an SD card. An external trigger would signal the start/end of recording.
The Arduino Ethernet Shield 2 may not be appropriate since it's documentations states:
Note that because the W5500 and SD card share the SPI bus, only one at a time can be active
Thank you for your suggestion. I forgot to mention in my original post that the data stream will be wired ethernet or RS485, so wifi is not an option for this project.
the raspberry-pi-3-model-b has built in Ethernet, SD card and a powerful 64bit processor so for data logging should have no problems. In the end writing speed to the SD card would probably be the limiting factor- how much data per second and how much data are you looking at?
Regarding RS485 I have never used RS485 with a pi but have used 3.3V TTL serial so should be no problem. connecting a TTL-RS485 converter.
Thank you for that information. I'll look into Raspberry Pi and conveeters.
The data rate is expected to be ~ 200 to 500 kilobits/s, and the total recording time ~ 200 seconds.
I don't know if something like an Uno would be fast enough for that, but a Mega should be able to handle it. But the biggest problem may be the card itself. SD cards deal with data 512 bytes at a time, which is the size of a sector. And sometimes there is a significant delay on a sector write because the card's controller has to erase sectors before writing, or even move data around if it does wear leveling. And if the file system is involved in the write, then you have to deal with updating the directory entry and the FAT tables. So while the average write speed may be ok, you might need a very large buffer to carry you through these delays without losing data.
There are ways to reduce your exposure to these problems, the best of which I've found to be illustrated in the LowLatencyLogger example in the SdFat library. Basically, he bypasses the file system altogether, and does sequential low-level sector writes to the data portion of a pre-defined, pre-erased file. Only at the end are the directory entry and FAT tables modified to reflect what was actually written. And I believe he saves to the card in binary, and has routines to convert that to a CSV readable file later when time is not pressing.
What it ususally means is that you select the Ethernet shield when you send/receive data and the SD card when you want to read/write. Libraries handle that for you. If you post the link to the shield, somebody might be able to verify.
And with any processor that is not multi-core and does not have multiple SPI buses, it will never be truely 'the same time'.
50kbytes/sec is a lot of data/sec where is it comming from?
over 200 seconds that is 10Mbytes in total
never attempted 10Mbyte arrays on a pi - works OK on a Win10 PC using Java - will try on a pi
edit: ran following program on a Raspberry Pi 3B
// create 10Mbyte array - fill with data and print some elements to check
public class Array10Meg {
public static void main(String[] args) {
int p[]=new int[10000000];
for(int i=0;i<10000000;i++) {
p[i]=i;
}
for(int i=0;i<010;i++) {
p[i]=i;
System.out.println(p[i]);
}
for(int i=9000000;i<9000010;i++) {
p[i]=i;
System.out.println(p[i]);
}
}
}
// create 10Mbyte array - fill with data and print some elements to check
import java.io.*;
public class Array10MegWrite {
public static void main(String[] args) {
try {
// create a writer
FileOutputStream fos = new FileOutputStream(new File("output.dat"));
BufferedOutputStream writer = new BufferedOutputStream(fos);
int p[]=new int[10000000];
// fill array with data
for(int i=0;i<10000000;i++) {
p[i]=i;
}
// write array to a file
for(int i=0;i<10000000;i++) {
writer.write(p[i]);
}
writer.close();// close file
fos.close();
}
catch (IOException ex) {
ex.printStackTrace();
}
try {
// now read file back
int p1[]=new int[10000000];
FileInputStream fis = new FileInputStream(new File("output.dat"));
BufferedInputStream reader = new BufferedInputStream(fis);
int ch, i=0;
while ((ch = reader.read()) != -1) {
p1[i++]=ch;
}
System.out.println("read " + i + " bytes");
// and print contents
for(i=0;i<010;i++) {
p1[i]=i;
System.out.println(p1[i]);
}
for(i=9000000;i<9000010;i++) {
p1[i]=i;
System.out.println(p1[i]);
}
}
catch (IOException ex) {
ex.printStackTrace();
}
}
}
