100 values from a device ... how to best work with them (class, struct)?

[MaxG]

I have a controller (UNO) that pulls data via Modbus, and then publishes these as MQTT messages.

Instead of sending 100 values every 10 seconds, I want to send only the changed values.

I didn't want to use send 100 variables, and do a 'if changed send' condition, but thought of using a struct, and send only the changed values.

The idea is to store the previous value, a factor, whether the value was sent, and whether it is signed or not.

Is this the right setup?

struct modbus_data
{
    // register_address = array index + 8000
    bool b_publish_or_not;
    bool b_signed_or_not;
    uint8_t factor;                             // 0 = none, 1 = 0.1, 2 = 10
    uint16_t previous_value;
};

struct modbus_data arr_modbus_data[NUM_MODBUS_ARR_ITEMS] =
{
    // publish, signed_or_not, factor, previous_value
    {1,1,1,0}, {1,1,1,0}, {1,1,2,0}, {1,1,1,0}, {1,1,1,0},
    {1,1,2,0}, {1,1,1,0}, {1,1,2,0}, {1,0,0,0}, {1,1,1,0},
    {1,1,1,0}, {1,1,1,0}, {1,0,0,0}, {1,1,2,0}, {1,1,2,0},
    {1,1,2,0}, {1,1,2,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,1,2,0}, {1,0,0,0}, {1,1,2,0}, {1,1,2,0}, {1,1,1,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,1,2,0},
    {1,1,2,0}, {1,1,2,0}, {1,1,2,0}, {1,0,1,0}, {1,1,2,0},
    {1,1,2,0}, {1,1,2,0}, {1,1,2,0}, {1,1,2,0}, {1,1,1,0},
    {1,1,1,0}, {1,0,1,0}, {1,0,1,0}, {1,0,1,0}, {1,0,1,0},
    {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0}, {1,0,0,0},
    {1,0,0,0}, {1,0,2,0}
};

And is this the right approach to update the values?

// get sign: value signed = 1, unsigned = 0
bool b_signed_or_not = arr_modbus_data[array_id + i].b_signed_or_not;

// get the stored value
uint16_t stored_value = arr_modbus_data[array_id + i].previous_value;

I haven't worked with these structs at this complexity before.

1. Is this the wya to do it?
2. Is there a better way to do this?
3. Would a class be a better approach (getters/setters)?

Any hints appreciated.

[DaveX]

It seems like you'd then need to send 400 values.

It is unclear how you know which value is sent in this scheme, compared to if you send the full set, you know that the nth value means a specific thing.

[UKHeliBob]

You could iterate through the array, determine which items have changed and send whatever data you need but include the array index as part of the data so that the receiving system knows what the data relates to

[gcjr]

@MaxG

I want to send only the changed values.

how do you indentify the changed value? which of the 100?

[MaxG]

@DaveX

It is unclear how you know which value is sent in this scheme

@gcjr

how do you identify the changed value? which of the 100?

I haven't shown this , but it would be "compare new parameter (say 5), against the array value (previous_value), if it is different, then send".

I haven't shown this, because I am wary whether my approach to this problem is correct or not... the main question, is this how I should use a struct, or is a class a better option?

So, again, I showed where to store it, how to save and how to query the value, but not the compare... which is a simple if (received_value_unsigned != stored_value)... and seemingly a minor issue.

Looking at this again, I can do away with bool b_publish_or_not;. Simply compare new to stored, and send, if changed... but I think I wanted to update the struct with all values first, and then have one function that publishes what has changed, and reset this flag for those.... and loop to do it all again; as in:

• read vlaues
• store changed ones and change flag
• iterate through struct and publish
• repeat

[UKHeliBob]

I think what @gcjr was getting at was not how you identify which values have changed in order to send them but how the system that receives the changed data knows what the changed value relates to

If that was not his question then I would like to know anyway

[gcjr]

how the data is stored and the format of a message are separate. it would make sense that if data #5 is updated in a message, that the message would include the value of the data and an ID, 5.

it's also conceivable that a single message includes multi updates

how the data is stored should presumably be in such a way that minimizes the logic both to update the data as well as to determine when to send an update.

@MaxG

    // publish, signed_or_not, factor, previous_value
    {1,1,1,0}, {1,1,1,0}, {1,1,2,0}, {1,1,1,0}, {1,1,1,0},

wouldn't it make sense to using single letter enums instead of integers to indiate the parameters. signed/unsigned, S/U, publish/unpublish, P/U, F for factor and '_' for not.

[noiasca]

@MaxG

Instead of sending 100 values every 10 seconds, I want to send only the changed values.

Why not just store the last sent value.
When the new measured value differs from the last transmitted value - transmit it.

no need for published, signed, factor ...

[jremington]

If you have a "old" array of 100 values, and collect a new one, then it is trivial to step through the two arrays, compare values one by one, and send only the changed values (each with its index, as already suggested).

But if more than half the array values change, you end up sending more than 100 items.

[MaxG]

@UKHeliBob

I think what @gcjr was getting at was not how you identify which values have changed in order to send them but how the system that receives the changed data knows what the changed value relates to

// the offset from the first register address to arrive at the array index for
// the arr_modbus_data[]
#define ARRAY_ID_OFFSET 8000

The received parameters are in order from 0 to 107, or 8000 to 8107.
I query sections of the data, like 0 to 10, 11 to 25, 25 to 44... etc.

Hence, I use the parameter ID - 8000 as array ID.

The data received or pulled from the device, is always queried in sequence.
I then compare the received value to the stored, if different, then store it... as described earlier.

@noiasca

Why not just store the last sent value.

Yes, what I have hoped I had described

• pull all 107 values in order
• compare, store new value and set 'publish' flag
  eventually grab the struct and publish the values with a 'publish' flag, and reset this flag
  repeat

My question remains, is this a good approach, or am I over complicating this, or is there a better way. I stated in other recent posts that I am learning C++, or rather uplift my skills from using basic conditionals to structs and classes. Hence, these potentially silly questions.

[MaxG]

@jremington

If you have a "old" array of 100 values, and collect a new one,

sorry, no... I have one data struct:

• receive a new value
• compare new against old, and store new if required (and set updated flag)
• then send any updated ones (and reset updated flag)

[UKHeliBob]

@MaxG

I have one data struct:

You do at the moment but you are asking whether that is the best way to hold the data and maybe it isn't

[jremington]

We are discussing the same basic approach. I don't see any issue, except the for mentioned case when more than half the values change.

[gcjr]

what's your message format? (maybe this will get an answer)

[DaveX]

I'm sorry, from the #1 question, I though you were sending a single message with the entire 100 values every 10s. But now I think perhaps you mean that you were sending 100 messages every 10s.

Maybe you are asking if an array of structs is a good pattern for storing a set of 100 records of data? I'd say yes.

And yes, these look like the proper way of accessing the members of struct within the array (given that arr_modbus_data[] is in scope.)

@MaxG

is this the right approach to update the values?

// get sign: value signed = 1, unsigned = 0
bool b_signed_or_not = arr_modbus_data[array_id + i].b_signed_or_not;

// get the stored value
uint16_t stored_value = arr_modbus_data[array_id + i].previous_value;

[MaxG]

@UKHeliBob

You do at the moment but you are asking whether that is the best way to hold the data and maybe it isn't

Sure... I hope nobody sees my replies as defensive or aggressive... I am clarifying, and get it that I as the poster have a full-picture view, but may not share enough for others to get the hang of it...

So most certainly, my question remains: is what I am doing best practice?! I'd like to do a good job, but understand I lack the skills to do so.

@jremington

when more than half the values change.

I actually have to check how many values are changing...

---

I reckon I have to provide more context: The device is an inverter charger that sends solar, battery, grid an state parameters, in total 107 parameters.

Here some code where I read a section of the data with my current approach, not using the struct as a data store, but processing sequentially all 107 parameters (changed or not).

The crux is, sending 107 messages every ten seconds all day results in one million messages. I felt this to be excessive, hence, wanted to reduce the traffic to changed values only, whne I thought about this struct approach.

/*
 * ----------------------------------------------------------------------------
 * @brief Read a block of Modbus register, here mainly AC Load values
 * ----------------------------------------------------------------------------
 */
void read_ac_load ()
{
    static const uint16_t start_address = 0x1f44;   // 8004
    static const uint8_t number_of_registers = 4;   // 8007
    static uint8_t result = 1;

    result = modbus.readHoldingRegisters(start_address, number_of_registers);

    if (result == modbus.ku8MBSuccess)
    {
        if (1 == OUTPUT_VALUES_VIA_SERIAL)
        {
            Serial.print(F("AC_Load_Power...........: "));
            Serial.println((int16_t)modbus.getResponseBuffer(0x00) * 10);

            Serial.print(F("AC_Load_Voltage.........: "));
            Serial.println((int16_t)modbus.getResponseBuffer(0x01) / 10.0, 1);

            Serial.print(F("AC_Load_Frequency.......: "));
            Serial.println((int16_t)modbus.getResponseBuffer(0x02) / 10.0, 1);

            Serial.print(F("AC_Load_Energy..........: "));
            Serial.println((int16_t)modbus.getResponseBuffer(0x03) / 10.0, 1);
        }

        if (1 == OUTPUT_VALUES_VIA_MQTT)
        {
            g_register_value_signed = (int16_t)modbus.getResponseBuffer(0x00);
            g_register_address = start_address + 0;
            data_publish_int(g_register_address, g_register_value_signed, 10);

            for (uint8_t i = 1; i < number_of_registers; i++)
            {
                g_register_value_signed = (int16_t)modbus.getResponseBuffer(i);
                g_register_address = start_address + i;
                data_publish_int(g_register_address, g_register_value_signed, -10);
            }
        }
    }
    else
    {
        decode_error(result, start_address);
    }

    return;

}   // read_ac_load ()

@DaveX

I'm sorry, from the #1 question, I though you were sending a single message with the entire 100 values every 10s.

No need to be sorry...

As stated before, I am not used to constructs like arr_modbus_data[array_id + i].b_signed_or_not; You know, things having a method)... I am learning, so while (so it seems or you say) it does what it is supposed to do, I haven't grasped this yet as being a good way of doing things.

So based on your response, thank you, I will update my other functions to reflect this struct approach.

[DaveX]

This thread:

Flashing multiple LEDs at the same time

Introduction A common question is to ask how to flash 2 (or more) LEDs at the same time. Typically the person asking will be using delay and will be experiencing the main problem with delay: it blocks and stops anything else from happening. This tutorial contains several different approaches to the problem of flashing 2 or more LEDs at the same time with independent timing for each LED. The basic principles for each can all be adapted for other situations requiring multiple timers working inde…

... has a few examples of doing simple repetitive things different ways, with some using arrays of structs, arrays of classes, etc.

[jremington]

@MaxG

The device is an inverter charger that sends solar, battery, grid an state parameters, in total 107 parameters.

How many of those values are actually useful, and of interest to you? I imagine that only a small fraction of them is actually worth saving.

[MaxG]

@jremington

How many of those values are actually useful, and of interest to you?

Believe it or not, 94 of them

To give you an idea:

char * decode_battery_charging_status (uint8_t charge_mode);
char * decode_ac_status (uint8_t status_code);
char * decode_attention_required (uint8_t attention_code);
char * decode_generator_status (uint8_t status_code);
char * decode_generator_reasons(uint8_t reason_code);
char * decode_charge_mode(uint8_t charge_mode);
char * decode_button_mode(uint8_t button_mode);
void data_publish_int (
    uint16_t g_register_address,
    int16_t g_register_value_signed,
    int8_t scale
    );
void data_publish_uint (
    uint16_t g_register_address,
    uint16_t g_register_value_signed,
    int8_t scale
    );
void process_value (
    uint16_t register_address,
    uint16_t received_value,
    uint8_t signed_or_not,
    uint8_t factor
    );
void read_battery();
void read_ac_load();
void read_ac_source();
void read_inverter();
void read_controls();
void read_inputs_outputs();
void read_solar_hybrid_priority();
void read_ac_source_status();
void read_generator_status();
void read_attention_codes();
void read_various_1();
void read_solar();
void read_various_2();
void read_various_3();
void write_single_register (
    uint16_t g_register_address,
    boolean b_is_int_signed,
    uint16_t g_register_value
    );
int16_t charArrayToInt (char *arr);
void set_schedule_bit (
    uint8_t schedule_id,
    uint8_t state,
    uint8_t schedule_state_variable
    );
void print_section_millis (uint8_t address_section);
void enforce_hybrid_schedule();
void read_and_write_registers();

I am almost of grid; the grid is just another 'generator'...
Lots of interdependencies between PV generation, is the battery full or not, pump water, export, charge car, charge mower, charge digger... status and error messages; the latter will trigger e-mail, SMS or other means of alarming. What I can do away with is serial numbers, MAC, IP address, and a handful of other static ones.