Networking Arduinos to a database for motor control via Ethernet or Wi-Fi?

So, brief overview of my "vision"...

I want to have about 20 Arduino Megas controlling 10 stepper motors each. The motors will not be in sync & each group of 10 will be separated by ~8 feet. The commands for their control would be coming from a server computer with a database setup to contain all of the relevant info such as when to spin, how many steps, etc. This info would be sent to the Arduino's code & executed over Ethernet. General flow would be:

(From Database) When to turn, how far to turn, which motor to turn, etc ->
(Backend code) Receives these variables (5 seconds, 200 steps...) & does whatever is needed ->
(Variables) -> Sent over Ethernet/Wi-Fi to the Arduino Mega with an Ethernet/Wi-Fi shield.
(Arduino) Sets the required outputs on each pin to control motors

So, there are multiple questions I've been answering over the last few weeks for myself, but the biggest ??? for me right now is the networking aspect. If I were to use this setup with an Ethernet/Wi-Fi shield, am I only able to control the pins physically attached to the shield? So, using a Mega, most of my IO would be useless. Or does the shield enable control & feedback of & from all the pins on the board (not "reserved" for the shield's use)?

~`~

Extra info below, for further consideration

~`~

I plan on using the Apache web server for my web-app. The database will be handled with MySQL. I'm planning on doing a majority (hopefully all) of my back end coding with Python.

I don't have much (any) experience with networking, which is why some of my questions may seem either very basic, or may not even make sense. I'm learning as I go for this aspect, I feel pretty comfortable with the rest of the project though. I've done a lot of searching & have found some very undetailed tutorials that I've been trying to piece together.

Ethernet
the WiFi interference of 20 stations would be a problem

the shield uses only SPI pins (50, 51, 52) on the ICSP header and pin 10. (and pin 4 if SD card is inserted). pin 53 must be an output for SPI

To answer your "shield contorl pin?" question, NO. Your arduino still controls all those pins.

You should describe your system in more details, such as how far these arduinos are separated so whether they can all be connected to a central computer like a raspberry pi, ridding the need for individual arduino + individual wifi shield. You can run long cables at 100ft (say 30m) with proper adapters between your arduinos and raspberry pi so you don't need individual wifi shields. The less the wireless connections, the less issues you will see. The raspberry pi can even run your data base and tell which arduino to do what without having to be told by a separate server.

Juraj:
Ethernet
the WiFi interference of 20 stations would be a problem

the shield uses only SPI pins (50, 51, 52) on the ICSP header and pin 10. (and pin 4 if SD card is inserted). pin 53 must be an output for SPI

So, when you say only those pins are used, does that mean only those pins can be controlled via Ethernet? Or all the pins but those can be controlled via Ethernet?

milanschreiber:
So, when you say only those pins are used, does that mean only those pins can be controlled via Ethernet? Or all the pins but those can be controlled via Ethernet?

the pins of Mega are controlled by Mega (ATmega 2560). the shield doesn't change it. the Mega communicates with the shield over SPI (pins)

liuzengqiang:
To answer your “shield contorl pin?” question, NO. Your arduino still controls all those pins.

You should describe your system in more details, such as how far these arduinos are separated so whether they can all be connected to a central computer like a raspberry pi, ridding the need for individual arduino + individual wifi shield. You can run long cables at 100ft (say 30m) with proper adapters between your arduinos and raspberry pi so you don’t need individual wifi shields. The less the wireless connections, the less issues you will see. The raspberry pi can even run your data base and tell which arduino to do what without having to be told by a separate server.

The arduinos will be about 8 feet apart each, 2 columns, 4 rows. I would have a central (windows) computer for each bank of 8 Arduinos. The maximum length I should have to go is about 50 feet. Unfortunately, I’m stuck using Windows, rather than RBPi which I would prefer. I had assumed 50 feet would be too far for USB.

Juraj:
the pins of Mega are controlled by Mega (ATmega 2560). the shield doesn’t change it. the Mega communicates with the shield over SPI (pins)

Ahhh, I think that clears it up for me & explains why I couldn’t really find info on this. Does this mean the shield essentially behaves as a replacement for the USB port for communication? So, without the shield, I can send code over the USB, but with the shield, I am able to send code over Ethernet?

milanschreiber:
Ahhh, I think that clears it up for me & explains why I couldn't really find info on this. Does this mean the shield essentially behaves as a replacement for the USB port for communication? So, without the shield, I can send code over the USB, but with the shield, I am able to send code over Ethernet?

so this will be hard.

the TCP/IP network communication is not so simple as Serial (over USB). it will not replace Serial.

the Ethernet (or WiFi library) wraps the TCP/IP networking concept of sockets. the client creates a socket and connects it to a TCP server. the server listens for incoming connection and creates a sockets on server\s side. then the two sockets are connected and can exchange data the same way as Serial. the TCP protocol cares about delivering the packets between sockets. (there is also UDP protocol which sends messages and doesn't care if they are delivered)

sketch upload over Ethernet is possible, but requires to change the bootloader of Mega.

Juraj:
so this will be hard.

the TCP/IP network communication is not so simple as Serial (over USB). it will not replace Serial.

the Ethernet (or WiFi library) wraps the TCP/IP networking concept of sockets. the client creates a socket and connects it to a TCP server. the server listens for incoming connection and creates a sockets on server\s side. then the two sockets are connected and can exchange data the same way as Serial. the TCP protocol cares about delivering the packets between sockets. (there is also UDP protocol which sends messages and doesn't care if they are delivered)

sketch upload over Ethernet is possible, but requires to change the bootloader of Mega.

I think I'm getting closer to understanding, thanks for pointing me in the right direction! I just read up on sockets. So, from what I understand... The Arduino (client) will have its sketch that will request a connection to the server. I had initially planned on the server side sending the variables to the Arduino's code when needed.

So for example, if I had in my table: Arduino 2's motor 6 is to turn clockwise 40 steps (in column X) after 600 seconds (in column Y), then return home after 200 seconds (in column Z), I'd have some code along the lines of:

Delay(Y)
ar2_Stepper6.step(X).
Delay(Z)
ar2_Stepper6.step(-X)

...located on the Arduino.

Would I have to request the value from the database after each run of code to ensure that the values are still the same?

milanschreiber:
I think I'm getting closer to understanding, thanks for pointing me in the right direction! I just read up on sockets. So, from what I understand... The Arduino (client) will have its sketch that will request a connection to the server. I had initially planned on the server side sending the variables to the Arduino's code when needed.

So for example, if I had in my table: Arduino 2's motor 6 is to turn clockwise 40 steps (in column X) after 600 seconds (in column Y), then return home after 200 seconds (in column Z), I'd have some code along the lines of:

Delay(Y)
ar2_Stepper6.step(X).
Delay(Z)
ar2_Stepper6.step(-X)

...located on the Arduino.

Would I have to request the value from the database after each run of code to ensure that the values are still the same?

the Arduinos can connect to server on PC and stay connected. the PC will manage 20 open sockets and send commands over them to Arduinos. or every Arduino can run a TCP server and the PC connects to them.

only new commands need to be send. you can make your Arduino sketch to execute a received sequence or repeat it specified times or forever

Use one USB-CAT-5 adapter. You can get over 100ft.

Locally you can use two usb hubs daisy chained to get all 8 together. Say this one:

It has 7 ports. You use one of the ports to daisy chain a second hub. You know have 13 ports. The hubs can be powered with included ac adapters so it won't run undervoltage with 8 arduinos.

Then use one USB-CAT-5 adapter on the main hub's upstream usb connection and good quality CAT5 or CAT-6 cable to your PC.

Do you think this is maybe worth trying, with hubs and adapter, instead of equipping every arduino with a wifi shield (lots of money) and try to figure out how to talk over wifi?