Esp32 large data collection,

Projects Programming
1

hi. i am sending data to server thorugh LTE. by using esp32 and LTE Module. data about 50 veriable like int, array..etc. in one packet , using sprintf funtion all data convet to one single array then send via http post. and its work perfectly fine. now i m facing two hardle , when LTE not avaliable store packet and after signal available send all data. issue is:
how to store hundreds data packate in esp32 , and second how to send large data in one time over http server connection.

sprintf(body,"[{\"id\":\"%s\",\"sb\":\"%s\",\"sc\":%d,\"oa\":%d,\"ob\":%d,\"oc\":%u,\"od\":%d,\"oe\":%d,\"of\":%d,\"og\":%d,\"oh\":%u,\"oi\":%u,\"oj\":%u,\"ok\":%u,\"ol\":%d,\"om\":%d,\"on\":%d,\"oo\":%d,\"op\":%d,\"oq\":%d,\"or\":%d,\"os\":%d,\"ot\":%d,\"ou\":%d,\"pa\":\"%s\",\"ax\":%d,\"ay\":%d,\"az\":%d,\"am\":%d,\"at\":%d,\"ga\":%0.6f,\"gb\":%0.6f,\"gc\":%d,\"gd\":%d,\"ge\":%d,\"gf\":%d,\"ha\":%d,\"hb\":%d,\"hc\":%d,\"hd\":%d,\"e1\":\"%s\",\"e2\":%d,\"e3\":%d,\"result\":\"%s\"}]",        
imei,sim_iccid_arr,int(gsmRssi),int(fuelStatus),int(engineload*100),int(coolantTemp),int(sf1*100),int(lf1*100),int(sf2*100),int(lf2*100),gasPressure,manifold_press,rpm,vehicle_speed,int(timing_adv),int(intakeTemp),maf,int(throttle*100),int(fuelT),int(hybridBattRem),int(odometer),int(hybrid_Bvolt),int(fuel_percentage*100),int(veh_bat*100),vin_arr,int(angleX*100),int(angleY*100),int(angleZ*100),int(mag*100),int(tempT*100),GPSlatitude,GPSlongitude,int(GPSspeed*10),int(GPSaltitude*10),int(accuracy*10),int(usat),int(carTrip),int(device_plug),int(device_volt*100),int(vMCheck1),cantype,int(ver),int(otaError),rxResult);

this is my one packet.

2

use an FRAM to store your data until it is ready to transmit
if transmitting large data files use FTP?

3

Seems a JSON formatted string...

You could simply store as a file inside ESP filesystem.

4

i dont want use extra chip bcz basically i not need non volatile memory as much. and yes after i get large data file i try to send via FTP. first issue how to get 100s of packet smartly,

5

@cotestatnt kindly share some example or detail, and yes its JSON frame.

6

You can find complete examples in the official ESP32 Arduino Core.
If you need to work with JSON I recommend you the library ArduinoJSON that allows you to serialize / deserialize data directly on the file.

Here a small wokwi example based on the LittleFS_test.ino included in ESP32 Arduino core, with the addition of ArduinoJson.

7

The ESP32 comes with the OS freeRTOS which has QUEUES that can be used to store data into.

8

hey thankz for share example its really help full for me, i try this and i convert all my data on Json format. my json packat data size almost 800 to 900 byte. i wonder who i store hundares of this kind of packet , kindly give me some way.?

11
11910×89 3.06 KB

9

hey thank for guiding me ,

void Demo_Task(void *arg)
{
    char txBuffer[50];
    queue = xQueueCreate(5, sizeof(txBuffer)); 
    if (queue == 0)
    {
     printf("Failed to create queue= %p\n", queue);
    }

    sprintf(txBuffer, "Hello from Demo_Task 1");
    xQueueSend(queue, (void*)txBuffer, (TickType_t)0);

    sprintf(txBuffer, "Hello from Demo_Task 2");
    xQueueSend(queue, (void*)txBuffer, (TickType_t)0); 

    sprintf(txBuffer, "Hello from Demo_Task 3");
    xQueueSend(queue, (void*)txBuffer, (TickType_t)0);  

    while(1){
        vTaskDelay(1000/ portTICK_RATE_MS);
    }
}

i think you point me this method . i convert my data in json format array, and this array size is 800 to 900 byte and i trying to store hunders of array, can this QUEUES menthod on esp32 handle this?

10

Yes for shure.
Depending of your specific board, but usually ESP32 dev-boards come with a 4MB flash memory.

Some of this is reserved for firmware, but you can use the remaining as you need.
You can choose the partitioning schema also in the Arduino IDE

image

11

rather than storing Json store the original binary sensor data
convert to Json when you extract the data for transmission
usually when transmitting large volumes of data I transmit the original binary and convert to the required display format at the server

12

partition_size_12-2022_new
partition_size_12-2022_new879×523 34.8 KB

i use this partition with my 16mb esp32 chip with ota serivce my bin file size is alomst 1.2mb,

13

but i think first we need to store in memory . my first issue is store large data in local memory efficiently then second step is sending on server when gsm get signal, i am currenly on first step.

14

The freeRTOS reference manual

15

the idea is

  1. sensor binary data stored into memory

then to transmit over GSM

  1. read binary data from memory > convert to Json > transmit Json over GSM

the problem is Json is very verbose and takes up a lot of storage (and communications bandwidth)
storing the original data in binary will probably reduce the storage requirement by an order of magnitude

16

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.