Practical/common approaches to 'using' threadsafe functions for simple shared variables?

Projects Programming
1

Practical/common approaches to 'using' threadsafe functions for simple shared variables?
(By virtue of being at best rusty, I'm basically an Arduino/C++ beginner)

An evolving scenario:

  • I start of with a simple problem: I have different functions that will want to access the same variable, so lets just assume a global variable.
  • Now lets say 'some' these functions will actually be run on a loop inside a thread/task (in my case ESP32 dual core & FreeRTOS, but I'm I'd be interested in both platform specific comments and more general answers).
  • But not 'all' of these functions will be tasks, some might just reside in setup() for a while.
  • What this means is that I can't just use a global variable any more. The easiest thing to do is probably just to accept that I need to use some kind of threadsafe, specifically implemented tool like a semaphore(mutex?) or queue for the purpose, or some such. Most of these are platform/implementation specific API offerings, but the concepts they offer are fairly generic.

Now I've spent days looking into this, googling it, etc... and the more I look the further away I get from an understanding of a 'practical' approach. Perhaps literally no two examples agree, or build up a coherent pattern, as to how one would actually code with with these tools, in a way that wasn't likely to go horribly wrong because of one mistake (a mistake that would mean your code wasn't threadsafe, slips straight past the compiler, and only ever crops up in run time in the most 'impossible' of circumstances to even begin to debug the cause of.)

Lets assume that this would-be-global is very basic, but not super basic, so lets assume its a C++ string, or any abstracted platform specific data structure (so we can't be too sure of the under-the-hood details, we don't want to have to be and that's the whole point.) So as far as I know there is no point in relying on the 'atomic::' class as an option.

'Realtime' performance is 'not' 'critical', nor is space really, but lets at least follow enough good practice that we don't utterly murder the scheduling.

Lets assume we were to use a semaphore to gatekeep that our variable is only ever accessed appropriately.
Now, looked at naively we could implement use of that semaphore in every task that wants to apply to work with our would-be-global variable, but one misstep, and we are done for, we might as well not use it at all.

So it seems like there ought to be a way, or a tool, or a pattern, that uses in a semaphore in one place, or some other tool, and applies to accesses our shared variable through in place, so that we can check we've got our coding right once, and then move on, reasonably confident that we are good.
E.g:

  • a class with a private property (variable)
  • only methods provide public access to get and put to that private property
  • internally the semaphores are used one time in one collection of code, to ensure safe access to out variable.
  • perhaps our class should be an event oriented task itself?
  • Is there someway to use queues here instead?
  • What should I be returning where when and how to ensure that I don't just move my thread safety problem from one piece of code to another.

Now, I could keep looking at this (and I will) but it feels like I'm not getting any closer.
It seems like there ought to be a clear pattern for

  • abstract problem, abstract code
  • re-usably implementing thread safety in one place
  • its for a simple problem:- like 'some' global data structure/object, but its not a complex resource like protocolling with a shared hardware interface or some such.

Any help with this would be appreciated.
I know that thread and RT concerns aren't the most beginner stuff, but it seems like this is the most basic problem to solve when you start working with tasks/threads, and no 'final complete' examples ever seem to pull in the same direction.

Sorry this is so rambling. But that's the issue, my progress has just got ever more rambling not less.

Thanks.

2

Sorry, I don't follow your 'problem'. I built a genuine pre-emptive multi-process, multi-task system with six physical CPUs and shared memory. I don't recall doing anything like you mention, BUT any 'shared' variables went in shared memory using homemade linked lists, queues, etc.
It may be clearer if you posted some sample code, specifically of something that doesn't work, so you can get suggestions as to how to fix it.
My gut feel is there is no issue but I may be missing your point.

3

Protecting access to simple data items of 32 bits or less on the ESP32 is only a problem across cores.
The general rule is to use core 1 only for your application and leave core 0 for the radio stuff etc.
Your options, as you have identified, is to use the operating system supplied mutex or queue structures.

4

Hi sonofcy, thanks. Using any Arduino architecture core (not python please) please provide an example of how this threadsafe-shared-memory is setup and written/read please? Perhaps this is exactly what I'm not getting. Or maybe this is something platform/architecture/core specific?

As for a code example, I just can't get anywhere near code without seeing a problem having to turn 120 degrees, and throw it in the bin.

The problem is still best described by.

  • abstract problem, abstract code
  • re-usably implementing thread safety in one place
  • its for a simple problem:- multiple tasks all reading and writing to something which is like 'some' global data structure/object, but its not a complex resource like protocolling with a shared hardware interface or some such.

You certainly can't just declare and use a global variable, or any structure made from ordinary storage/memory datastructures/variables, because its not threadsafe. Its harder to work out if its going to be safe, than it is just to assume its not safe. (could be interrupted any time, data could be in a half complete state as its read form address 'a' up to 'b' > overwritten elsewhere, and then finally read from 'b'-'c', then if you start making flow decisions based on that you could find yourself in a race condition. There is no point in testing, these problems can manifest very rarely, but unless your ok with ("1+1 sometimes equals 'whatever'" then they are still fatal) That's just the stuff I know about, never mind the stuff I don't.) Its easier just to assume its not safe. But I don't think that's what your suggesting.

5

Thanks 6v6gt.
For a start we need to assume its not just 32bits.
Could be a C++ string (dynamic array) at the every least. Or some ESP32/arduino structure we would prefer to know only the high level about.
As for fixing affinity to a specific core, that's still very interesting, are you able to link to an Espressif article please. Although it makes sense as 32bits just sounds like one byte in memory. Not even necessarily a whole word (don't know how ESP32 manages memory). So I don't see how you can have written anything, if you interrupt it. Yeah you can write bits, but I've always assumed that was effected for you by rewriting whole chunks/bytes/words.

6

I think this is the thing though. I can accept that for something architectural there isn't some read higher level object/class/method for my specific need.
Although what I'm doing is so basic, I'd be amazed if there wasn't some one way of dong things, that meets my criterion, that isn't being used a full 40% of the time.

eg.
define Class, private property, methods, maybe a queue somewhere. declare an instance. use the instance naively, rinse and repeat.

7

The system I mentioned was created several decades before Arduino existed, in fact in an earlier century.
I am still waiting to see some sample code that exhibits the problem you think you have.

8

Oh I see. I get you. Yeah, "just write some tasks that naively share a global variable."
I will have to write that, don't already have it, I've kept doing 120s away form that but I can. Will take me a while as the outline I have at the moment is too simple, I need something that is likely to work for what might come. Will update later with such.
Thanks sonofcy.

9

Is your code arranged in the classic/normal 1 ino file, many cpp fies, and 1 h file included in every ino and cpp file that needs it using #pragma once? That way a global in the .h file is visible to all ino and cpp files/functions/procedures.

If you are attempting to use both cores of a multi core board so have 2 compile units, then I think adding extern to a variable in both compile units makes it the same piece of memory and accessible to both (NOTE: I have not tested this, just read about it)

10

I just realized I did not address your very valid concerns properly. In the system I was speaking of, data that was passed from one process/task to another went through the shared virtual memory queues, which did have thread safety built in. The other variables were of the store, once generally at startup and read-only by any process/task thereafter, so thread safety was not an issue.
I can tell you that the system ran 24/6 for 10+ years, processing billions of transactions, and no data error was ever detected. Our users were always the first to tell us if something didn't look right.

11

I am having trouble understanding your problem, are you overthinking it? The pattern is rather simple.

sharedVar of some type

Setup
.....
create mutex(); // as ready
....
create tasks ...
end setup

Task 1
...
getMutex(); // wait for sharedVar to be free
do stuff with sharedVar ...
freeMutex; // done with sharedVar
... more stuff if needed
end task1

Task 2
...
getMutex(); // wait for sharedVar to be free
do stuff with sharedVar ...
freeMutex; // done with sharedVar
....
end task2

See Examples->ESP32->freeRTOS->Mutex

I would suggest changing the random wait time to 1000ms to make it easier to see what is happening.

12

Hi sonofcy, below is some code that quite quicky demonstrates the problem. It reproduces the hazard every handful of seconds.

#include <Arduino.h>

const char* test1 = "hello from task 1!";
const char* test2 = "GREETINGS FROM TASK II";


// --- Global Variables ---
// This global variable will be subject to a race condition.
String global_message = "";

// A spinlock to protect the global_message variable within a critical section.
portMUX_TYPE string_spinlock = portMUX_INITIALIZER_UNLOCKED;


// --- Task Functions ---

// Task 1: Modifies the global_message string.
// String manipulation is not atomic and can be interrupted, leading to corruption.

void task_modify_string_1(void* pvParameters) {
    for (;;) {
        // Overwrite the global string with a new message.
        global_message = test1;
        Serial.print(".");
        vTaskDelay(pdMS_TO_TICKS(150)); // Yield control for 150ms
    }
}

// Task 2: Also modifies the global_message string.
// This task will race with task_modify_string_1, potentially causing
// non-thread-safe corruption if a context switch occurs mid-operation.
void task_modify_string_2(void* pvParameters) {
    for (;;) {
        // Overwrite the global string with a new message.
        global_message = test2;
        Serial.print(".");
        vTaskDelay(pdMS_TO_TICKS(150)); // Yield control for 150ms
    }
}

// Task 3: Monitors the global_message and prints if it's in an unexpected state.
// This task uses a critical section to ensure the check and print are atomic.
void task_monitor_string(void* pvParameters) {
    for (;;) {
        // Enter a critical section to prevent other tasks from interrupting.
        taskENTER_CRITICAL(&string_spinlock);

        // Check for a corrupted state.
        if (global_message != test1 && global_message != test2) {
            Serial.print("\nCORRUPTED MESSAGE DETECTED: ");
            Serial.println(global_message);
        }

        // Exit the critical section.
        taskEXIT_CRITICAL(&string_spinlock);

        // Pause for a moment to let other tasks run.
        vTaskDelay(pdMS_TO_TICKS(50));
    }
}

// --- Arduino Setup Function ---
void setup() {
    Serial.begin(19200);
    // Give some time for the serial monitor to initialize.
    delay(10000);

    Serial.println("Starting FreeRTOS tasks to demonstrate race conditions.");
    Serial.println("Observe the unpredictable output on the Serial Monitor.");
    Serial.println("------------------------------------------------------");

    delay(10000);

    // Create the three tasks with a priority of 1.
    // The scheduler will handle running them concurrently.
    xTaskCreate(
        task_modify_string_1,
        "StringTask1",
        2048,
        NULL,
        1,
        NULL
    );

    xTaskCreate(
        task_modify_string_2,
        "StringTask2",
        2048,
        NULL,
        1,
        NULL
    );

    xTaskCreate(
        task_monitor_string,
        "MonitorStringTask",
        2048,
        NULL,
        1,
        NULL
    );

}

// --- Arduino Loop Function ---
void loop() {
    // The main loop is empty as all functionality is handled by the FreeRTOS tasks.
}

Looked at naively , each task assigns its value to the variable (either it does or it doesn't). A third task beings a critical (uninterruptible) piece of code to examine tha variable check it matches only one of the two values being assigned. It should only output if it finds that is not the case.
But indeed it does so, because the nature of threads is that can can interrupt each other at any time, and the process that makes up a simple matter of assigning a string consists of many instructions.... so you get corruption, a value that should not exist if the high level instruction viewed naively - serially.

It can be 'quite' timing sensitive, so change the instructions in there and the problem can go away as the timing changes, you can even easily block many instructions from ever completing, but I got this quite quickly, and its quite streamlined.

CORRUPTED MESSAGE DETECTED: GREETIfrom task 1!
........................................................................................
CORRUPTED MESSAGE DETECTED: hello from task ASK II
..
CORRUPTED MESSAGE DETECTED: helETINGS FROM TASK II
........................................................................................
CORRUPTED MESSAGE DETECTED: hello fromFROM TASK II
..
CORRUPTED MESSAGE DETECTED: helETINGS FROM TASK II
........................................................................................
CORRUPTED MESSAGE DETECTED: hello from tas TASK II
..........................................................................................
CORRUPTED MESSAGE DETECTED: hello from tOM TASK II
..
CORRUPTED MESSAGE DETECTED: heEETINGS FROM TASK II
13

Sorry I can't relate to your example. The way we did things, all variables that needed to be 'touched' by two tasks went via OS specific shared virtual memory queues with built in hardware locking. Later generations on a different hardware platform had software locking that we had to manipulate.
All our tasks when started built an in memory list of variables MOSTLY not shared but if they were, the rule was all other tasks could only read. We were ready to implement a locking method to allow multiple writers but in 10 years never found a reason to do so.
I should explain that we did have the abiity to add data (like a com port) via a supervisory task that our operators were trained in BUT the code added the new data to the doubly linked list before connecting the new element to the list and sending a signal to the reader task.

I guess what I am saying perhaps in a poor way is that my design was different from yours and did not have your problems.

Remember, in my system, no memory in the task was exposed to other tasks, all shared variables were in shared memory outside the compile unit. We had a root pointer that had to be put in the link edit step of every process that provided a starting point to the doubly linked list that every task if needed had to use.

I hope I explained that well enough.

14

Take a look at std::atomic<>.
Here's a discussion: https://forum.arduino.cc/t/thread-safe-data-transfer-interlock-using-std-atomic-and-std-memory-order/1159080?u=gfvalvo

15

From a practical perspective, you’re over thinking this. The most common behavior is to wrap the shared resource (regardless of type) in a mutex.

The standard way to solve the problem of remembering to access it properly is to wrap it in a class with locking accessors as you mentioned.

Either is fine and works for 99% of applications.

16

As I attempted to point out in post 11, every task that accesses a shared variable need to do synchronization. Random non-synchronized accesses of shared variables leads to chaos.