Large datalogger w raspberry?

Hi Folks!

I’ve read plenty of topics about communication between Arduino MCU and Raspberry computers, but still wondering, which way I should go!

Background:
I currently designing prototype of product (“Large Datalogger”) which will read large amount of sensor data (about 40 sensors), which will be also stored to the database for further rprocessing etc.

I’m quite certain that pure Arduino based system is out of the question, and also pure Raspberry based system, handling massive amount of sensors (reading and storing) with Raspberry alone, ain’t gonna be easy solution neither.

So combination of maybe 2 Arduino Mega MCUs (slaves) with one Raspberry server (with rasbian etc) might be good way to go?

Plan
What I’ve read, I would try to do it via serial connection, using virtual ports for each sensor, which won't be used simultaneously but one at time. Even there is about 40 sensors, sample rate is quite low most part, so that should be ok? If I understand this correctly, we would have to make python script on raspberry to listen those virtual ports and store the data on right place in DB.

Question 1: Could you point out some flaws in my current plan, is there better ways to do it? I will be eventually hiring some help to make this happen, but I'm curious what is quck and optimal (developer friendly) way to do it?

Question 2: Could I make communication between Arduino Megas and Raspberry easier and more robust if I replace two Megas with Linux capable Arduino Yun board (https://wiki.linino.org/doku.php)? Using eth - eth OR wifi – wifi instead of serial port connection? If so, does that Yun board make handling sensors more difficult than using Megas?

I appreciate any suggestions!

arduino201:
I’m quite certain that pure Arduino based system is out of the question,
............
is about 40 sensors, sample rate is quite low most part,

The only sensible technical information you have given in your post is the number 40, and have described that as massive, which it isn't. You haven't even detailed what any of the forty sensors are, and I don't I dare ask what a virtual port might be, but I don't think there is anything you have said that suggests you cannot do this with a single Mega.

What are these sensors? How far apart are they? How do you read data from them?

Nick_Pyner:
The only sensible technical information you have given in your post is the number 40, and have described that as massive, which it isn't. You haven't even detailed what any of the forty sensors are, and I don't I dare ask what a virtual port might be, but I don't think there is anything you have said that suggests you cannot do this with a single Mega.

Sorry i didn't elaborate. Most of these are position sensors and frankly i dont know do i need linear/rotary encoders or could i use steppers (to give position information), it will be powered with pneumatic cylinders, . multiple rpm sensors, timers, air flow, air pressure etc. But it's good to know if you think one Mega is adequate.

Nick_Pyner:
I don't I dare ask what a virtual port might be

I meant software serial...

wildbill:
What are these sensors? How far apart are they?

All those sensors are about two meters apart.

With a Mega there is no need for SoftwareSerial - it has 3 spare HardwareSerial ports.

A stepper motor cannot provide position information.

A diagram of the project you are proposing to build would help us understand what is in your mind.

This Python - Arduino demo may be of interest. Also Serial Input Basics - simple reliable ways to receive data.

...R

It was not for me to suggest a mega is adequate, and certainly not on the basis of what you described, it's just that that nothing you said suggests it isn't. As things are, there is a clear limit on the number of rpm sensors, I believe it is six, and there are only sixteen direct analogue pins. On the other hand, you may have sensors that share pins.

Robin2:
With a Mega there is no need for SoftwareSerial - it has 3 spare HardwareSerial ports.

A stepper motor cannot provide position information.

A diagram of the project you are proposing to build would help us understand what is in your mind.

This Python - Arduino demo may be of interest. Also Serial Input Basics - simple reliable ways to receive data.

...R

Good information! Thanks for the links i'll check em out!

That stepper thing i think i was confused i saw somewhere thing (that looked like stepper motor w belt) traveling along system and providing position information, but that must have been rotary encoder.

Nick_Pyner:
It was not for me to suggest a mega is adequate, and certainly not on the basis of what you described, it's just that that nothing you said suggests it isn't. As things are, there is a clear limit on the number of rpm sensors, I believe it is six, and there are only sixteen direct analogue pins. On the other hand, you may have sensors that share pins.

Thanks, In this system there is just 3 rpm sensors.

For gigabytes of data storage, attach one tiny, cheap OpenLog to a hardware or software serial port and send the data using Serial.print().

jremington:
For gigabytes of data storage, attach one tiny, cheap OpenLog to a hardware or software serial port and send the data using Serial.print().

That's interesting suggestion... might be silly question, but could you elaborate does that SD card act like a buffer, if it can't handle that transfer real time basis or what? Intention is to get that sensor data stored in DB (raspberry).

does that SD card act like a buffer

No, a standard file type and file system readable on any computer.

Please read the product description.

Thanks, I will evaluate that option!