100+ sensors data per second over to cloud database

Using Arduino Project Guidance
1

Hi guys, i am fairly new to arduino and IOT. My main aim is to send data packets over wifi using arduino wifi shield to an online database.
The main problem hence is that the queries i have to run ranges in 10,000 - 20,000 per second. Now I have been doing some testing on ATMega 2560 board. Here is my setup for now ;

ATMega 2560 arduino board
Temp sensor
Arduino wifi shield
Cloud database hosted with a domain

The most i can send are 4 queries per second and that too start having connections issues like after 16 - 20 queries are send in succession.
Can anyone suggest a better hardware for a scenario where the queries or post http requests exceeds 10,000 per second.

Thank you

2

You have to check both ends of the transmission line, the problem (bottleneck?) may reside in the server.

If you use the String type, get rid of it before continuing! The 8 bit Arduino dynamic memory managment is known buggy.

What if you let the Arduino only create the data packets, without sending or touching the WiFi module in any way - how many packets per second can be prepared, with or without problems after some time?

For testing purposes you can connect a LED or piezo to an output pin, and toggle the pin after preparation of each packet.

3

I am moving to another server to check on that issue, setting up firebase for testing and one other online cloud platform for it.

Here is the code I am using to send data over internet connection :

    client.print("POST /test.php?t=");
    client.print(35);
    client.println(" HTTP/1.1");
    //client.println("Host:www.skytoursuk.com");
    client.println("Connection: Keep-Alive");
    client.println();
    Serial.println("Inserted");

I will check on creating empty data packets to check the transmission frequency of arduino board without the Wifi module.

4

The only sensor is for temperature.
Temperature of a mass can take minute to change one degree.

Getting the values to the internet is one problem
Getting storage with the quantity of data points you seek is another problem.
Many of the free plans limit one to less than 50,000 writes in a month. about 1 write per minute.

it might be easier to store the data locally, then connect the local device on the internet for remote access

5

Thank you for the prompt response, so lets see if I want to write it to a local database will it be possible to write 10,000 queries per second with the same board ? The frequency will be like 10 kHz or 10 ksps.

Or i need a new board to make it run on a local system too ?

I can write using serial, saw that ATMega can write 10,000 characters per second over serial but just to make sure or do I need to get a better board to even write the values manually ?

Thank you.

6

You need a big and fast storage medium, like a SD card. Find out what's the fastest on the market.

7

Hi,
What is the application that needs 10,000 samples per second?

What is the input quantity and magnitude?

Thanks.. Tom... :slight_smile:

8

Can you give a bit more information about the whole process ?
I know that it is not 100% pertinent to the solution, but it helps us understand the big picture.
the subject line says 100+ sensors.
would that mean you would have a data point with 2 or 3 pieces of information

sensor #, time of reading, reading of temperature , 10,000 total reading per second

byte, unsigned long, float ? x 10,000 ?

another question is the duration of the readings. is this a batch, runs for a short period, or is this continuous ?

9

DrDiettrich:
You need a big and fast storage medium, like a SD card. Find out what's the fastest on the market.

okay so SD card for like storing the data locally and then sync it online in the background ?

10

TomGeorge:
Hi,
What is the application that needs 10,000 samples per second?

What is the input quantity and magnitude?

Thanks.. Tom... :slight_smile:

hi,

The application are basically sensors that are monitoring internal parameters of an electrical turbine. The sensors are clocking 10,000 samples per second hence forming a wavelength with 10 KHz frequency.

The input quantity is a float value and is recorded 10 k times per second.

dave-in-nj:
Can you give a bit more information about the whole process ?
I know that it is not 100% pertinent to the solution, but it helps us understand the big picture.
the subject line says 100+ sensors.
would that mean you would have a data point with 2 or 3 pieces of information

sensor #, time of reading, reading of temperature , 10,000 total reading per second

Yes sorry, it is basically 100 different sensors each recording readings at 10 k samples per second. I can setup a different board for each sensor as long as i can send 10k samples per second either over to internet or to a common local controller.

and the data points are around 120 but each is recording same amount of data.

byte, unsigned long, float ? x 10,000 ?

If we just focus on 1 sensor first, than it is reading per 0.0001 sec and there are 10k total readings per second.
it will be float x 10,000

another question is the duration of the readings. is this a batch, runs for a short period, or is this continuous ?

and the batch will be run continuously, so can i have it stored somewhere locally and than synced with the online database.