Does Arduino need a real-time scheduler?

Interesting thread. I have been working on commercial hard- and soft- real-time control systems for a number of years. Since those terms are commonly misused and misunderstood here is an example of a hard-time system we did a few years ago: a mud-pump controller driving two pistons with 2-meter stroke, controlled with about 500 HP of hydraulic pumps (at 10,000 psi, the manifold pipe is something like 200 mm in dia), through a servovalve for each piston. Each piston moves by a polynomial equation and their outputs are summed through a check valve so that the flow is constant, and adjustable over a wide range. LVDTs with 2-meter stroke monitor each piston. Smooth startup and shutdown and several considerations of fail-safe behavior were used. We achieved this on an 8051 with external ADCs and DACs, and interrupt handlers (two levels, pre-emptive) with re-entrant libraries. We almost ran out of code space. In our case the time interval was not tiny, but we had hard completion deadlines. The piston profile came out of a lookup table (actually 1/4 of the entire profile, mirrored and phase-shifted as needed), the actual piston position was read, and that was all fed into a firmware PID routine which then calculated the next value for the servovalves. That had to be completed before the timer tick to update the valves. That timer got faster as the flow rate was increased. Then in the background was interrupt-driven serial I/O for the machine interface to control the mud pump and to monitor critical performance values (such as the current error value vs target position). This I/O had to be safely pre-empted by the piston routines with due consideration of atomicity of variables which might be in the midst of being updated by the control interface. In the end it all worked well, we delivered complete documentation and source code, and I have not heard of any problems. I went to Japan to help install and tune the system and then our part of the project was complete.

Where execution deadlines got tight I would set and clear some spare I/O bits and watch them on an oscilloscope or logic analyzer, something like High when active in a critical routine and Low when in safe extra time margin. As the flow rate ramped up you could see the bit transitions coming closer together. If they ever collided that would be potentially catastrophic since the control loop could not correctly function. We set the scope to trigger on the smallest safe interval and left it running overnight (without driving the actual pumps - simulated input). There was also simple instrumentation in the code to log deadline violations.

The project engineer was a delight to work with: a very experienced, practical guy. One example: we had to monitor the end-of travel limit clearance of the pistons (which weighed over 1000 kg as I recall) since we didn't ever want to drive them into their stops. No one knew what would happen if that occurred. Safe clearance was something like 5-10 mm, and it could not be observed easily by eye while running. His brilliant idea was to use an empty aluminum soft drink can in the gap and measure the crushed thickness. Worked great and no risk of harm to the pistons.

In this project a simple RTOS might have been a huge timesaver. But we also had to have timer interrupts and serial I/O interrupts and I am not aware of how easily RTOSes can fit with those. If it is possible to weave RTOS features into your own needed I/O hardware support, that could be helpful but also could get a bit complex.

I am using Teensy++2 on a project now and just got the ARM Teensy 3, and there is enough code and data space on these new Arduino devices. My concern would be: is the RTOS granular so we only need to use what fits our case, and can it co-exist with I/O device interrupt handlers?

Thanks

In this project a simple RTOS might have been a huge timesaver. But we also had to have timer interrupts and serial I/O interrupts and I am not aware of how easily RTOSes can fit with those. If it is possible to weave RTOS features into your own needed I/O hardware support, that could be helpful but also could get a bit complex.

My concern would be: is the RTOS granular so we only need to use what fits our case, and can it co-exist with I/O device interrupt handlers?

A good RTOS allows independent interrupt handlers. The three systems I ported to AVR/ARM Arduinos can be used as simple kernels with all I/O from the Arduino core or custom libraries.

I am starting to write optional I/O libraries that are integrated with the RTOS. Many existing Arduino I/O functions just do busy loops so putting a thread to sleep and waking it with an I/O done interrupt is more efficient.

If you need a highly optimized ISRs, systems like ChibiOS are designed to allow your ISR to run with no interference from the RTOS.

On ARM this works very well with priority interrupts. ChibiOS is designed to allow "Fast interrupts".

On some architectures ChibiOS supports a special class of “Fast Interrupts”, such interrupt sources have a higher hardware priority than the kernel so it is not possible to invoke system APIs from there.

The invocation of any API is forbidden here because fast interrupt handlers can preempt the kernel even within its critical zones in order to minimize latency.

The only hardware resource I uses is access to a timer interrupt for the system tick.

I piggyback on sysTick on ARM since both Teensy 3.0 and Due will have a hook. So no new hardware is used on ARM.

On AVR I am using the timer 0 compare A interrupt. I did this so no Arduino core code needs to be changed. I will release an optional patch to piggyback on the timer 0 overflow interrupt. This is already used for delay(), millis(), and micros(). Then not new hardware will be used on AVR.

I first started with Netduino, then switched to Arduino because of more hardware and components and also cheaper, I'm now fiddling with some RF components and I really really need Threads, I don't know exactly what threads are in .NET and what is the importance of RTOS. But I need precision timing, and execution from a couple of wirelessly enabled arduino units. Think of it as a wireless sequential light system, I messed with millis(), timers and lots of libraries, all are unpredictable and hang. Or just drift apart as time goes by in execution. I dunno maybe im doing something wrong, but when I used Netduino I could do this preety easily.

Anyway its explained like this:

The .NET Micro Framework CLR has only one thread of execution and owns all of the memory in the system. During normal operation, the CLR iterates in the interpreter loop and schedules managed threads using a round-robin algorithm, according to the priority of the threads involved. Each managed thread gets a 20-millisecond (ms) time quantum during which the runtime executes intermediate language (IL) code that belongs to the stack of the managed thread being serviced. When the managed thread goes to sleep or waits for a synchronization primitive, such as a monitor that cannot be acquired or an event that is not signaled, the CLR puts the thread in the waiting queue and tries to schedule another thread. In between scheduling managed threads, the CLR checks for any hardware events that might have been raised at the native device driver level. If an event occurred, the CLR tries to dispatch the event to the managed thread that requested it. A hardware event is associated with some kind of input/output (I/O) event, whether it be an interrupt-based GPIO event or a serial USART, USB, or I2C event.

The runtime interpreter loop starts by checking to discover whether a thread is ready for a time slice. If a thread is ready, the CLR schedules the thread for processing and tests to determine whether any hardware interrupt events require processing. If there are, the CLR processes the hardware interrupt event and moves the thread that handles the interrupt into the queue of threads that are ready to be processed. If no hardware interrupts have occurred, the CLR again tests to see whether a thread is ready for a time slice.

If the CLR determines that there are no threads that are ready to be processed, it sets a timeout timer and goes to sleep in order to conserve power. When the timeout timer triggers or an interrupt occurs, it wakes the CLR back up. At that time, the CLR determines whether an interrupt or the timer woke it up. If it was an interrupt, the CLR determines if processing the interrupt is required. If not, it sets the timeout timer and goes back to sleep. If it is the timeout timer that wakes the CLR, the CLR executes any pending completions and checks again for interrupts. If there are waiting interrupts, it goes through the normal interrupt processing procedures. If there are no waiting interrupts, the CLR sets its timeout timer and goes back to sleep.

Basic example with Netduino

namespace NetduinoApplication1
{
    public class Program
    {

        static Thread myThread = new Thread(new ThreadStart(myFunc));
        
        public static void Main()
        {
            myThread.Start();

            //suspend at some point, wait for event
            //myThread.Suspend();

            //resume at some point, wait for event
            //myThread.Resume();

            Thread.Sleep(Timeout.Infinite);
        }

        public static void myFunc() { 
            // do some stuff
        }

    }
}

My opinion: Arduino needs a better delay() function :grin:

The question RTOS or no RTOS does just heat up the discussion, but does not help the migration. This is my approach: I want a better delay() function - this is a feeling. If I introspect this feeling I recognize - it is a waste of energy for me if delay() is just doing busy waiting. - I want/I need to do several things in parallel, but delay() does not allow me to do it. - I do not want to throw everything away because of a bigbang RTOS solution, I want to go step by step from here to there.

As a seasoned software engineer I have seen again and again that migration is the key. Here are my suggestions:

Migration step 1: Introduce event handling

The Arduino "core library" has the loop() function. An event handling Arduino program will have one only statement in the loop() function, the do_event() function. The do_event() function has to look after the events and has to service the events. A useful vehicle to show that event handling is useful should be a keyboard scan library that provides its services within an event handling framework. The typical pocket calculator keyboard has arranged the keys between row and column lines. A 12 button keyboard needs 3+4=7 scan lines. The keyboard scan function is not trivial. Everybody expects to get: - key debouncing - key repeat (repetition) - 2-keys-rollover (Rapid typists may sometimes inadvertently press a key before releasing the previous one) - no electrical shortcircuit if two keys are pressed simultaneously To implement these functions, a timer is needed. The keyboard lib should provide a timeout functionality to the user. The function set of the keyboard lib is: button_register(): combine an event (key press) to a function (eventhandler) after(): after N milliseconds call a function every(): every N milliseconds call a function do_event(): check if there is a pending event and execute the eventhandler function. Needs to be called again and again within loop().

A simple Arduino program that fits nicely the event paradigma is a math trainer program. The Arduino displays a math question like "1 + 2 = ?" and the user should enter the correct answer within a time limit. In an event program there will be a keyboard event for every digit and a timeout event. If the user presses the digit "3", the button-3-event function will say "correct answer". The other button-event functions will say "wrong answer". And the timeout event-function will say "timeout".

Migration step 2: Create a event-driven LiquidCrystal lib

Everybody is using LiquidCrystal. But LiquidCrystal uses delay() and busy waiting is the nemesis to all real time programming. Thanks to the keyboard lib above, we have one-shot timer. And an event driven LCD output function does look like this: do_lcd() { switch (state) { case 1: // do state 1 stuff state=2; after(N, do_lcd); return; // set next state, use after to delay execution by N milliseconds, exit do_lcd() case 2: // do state 2 stuff state=3; after(M, do_lcd); return; // set next state, use after to delay execution by M milliseconds, exit do_lcd() } }

The busy waiting version of do_lcd() was: do_lcd() { // do state 1 stuff delay(N); // do state 2 stuff delay(M); } }

The do_lcd() function is ugly, but it is working without busy waiting. If all Arduino users are annoyed by event handling uglyness, they are ready for phase 3 of the RTOS brainwash program:

Migration step 3: Show that multi-threading does look better

Multi threading is dangerous. The worst thing you can do with busy waiting is to freeze everything up. The worst thing in multi threading is that you literally tear apart your machine because thread 1 says "go left" and thread 2 says "go right" as fast as time-division multiplexing can do. But, if you have multi-threading and a multi-threaded version of delay(), the do_lcd() function will no longer look ugly, and you are still out of busy waiting land:

do_lcd() { // do state 1 stuff delay(N); // the multi-thread version of delay() will switch to another thread or will wait in the operating system // do state 2 stuff delay(M); // the multi-thread version of delay() will switch to another thread or will wait in the operating system } }

I bet, everybody will be happy about a keyboard() lib. And they will learn that "fu**ing" event crap, because using a lib is better then writing a lib. And everybody who uses keyboard() will use after() and every(). And now the future is open: If you can live with the event programming uglyness, you never have to go to multi-threading land. Maybe you remember the X windows system, the GUI for UNIX and Linux? Well this was all done by using event programming, and it was done a long time before multi-threading was invented.

AndreAdrian: My opinion: Arduino needs a better delay() function :grin:

The question RTOS or no RTOS does just heat up the discussion, but does not help the migration. This is my approach: I want a better delay() function - this is a feeling. If I introspect this feeling I recognize - it is a waste of energy for me if delay() is just doing busy waiting. - I want/I need to do several things in parallel, but delay() does not allow me to do it. - I do not want to throw everything away because of a bigbang RTOS solution, I want to go step by step from here to there.

As a seasoned software engineer I have seen again and again that migration is the key. Here are my suggestions:

Migration step 1: Introduce event handling

The Arduino "core library" has the loop() function. An event handling Arduino program will have one only statement in the loop() function, the do_event() function. The do_event() function has to look after the events and has to service the events. A useful vehicle to show that event handling is useful should be a keyboard scan library that provides its services within an event handling framework. The typical pocket calculator keyboard has arranged the keys between row and column lines. A 12 button keyboard needs 3+4=7 scan lines. The keyboard scan function is not trivial. Everybody expects to get: - key debouncing - key repeat (repetition) - 2-keys-rollover (Rapid typists may sometimes inadvertently press a key before releasing the previous one) - no electrical shortcircuit if two keys are pressed simultaneously To implement these functions, a timer is needed. The keyboard lib should provide a timeout functionality to the user. The function set of the keyboard lib is: button_register(): combine an event (key press) to a function (eventhandler) after(): after N milliseconds call a function every(): every N milliseconds call a function do_event(): check if there is a pending event and execute the eventhandler function. Needs to be called again and again within loop().

A simple Arduino program that fits nicely the event paradigma is a math trainer program. The Arduino displays a math question like "1 + 2 = ?" and the user should enter the correct answer within a time limit. In an event program there will be a keyboard event for every digit and a timeout event. If the user presses the digit "3", the button-3-event function will say "correct answer". The other button-event functions will say "wrong answer". And the timeout event-function will say "timeout".

Migration step 2: Create a event-driven LiquidCrystal lib

Everybody is using LiquidCrystal. But LiquidCrystal uses delay() and busy waiting is the nemesis to all real time programming. Thanks to the keyboard lib above, we have one-shot timer. And an event driven LCD output function does look like this: do_lcd() { switch (state) { case 1: // do state 1 stuff state=2; after(N, do_lcd); return; // set next state, use after to delay execution by N milliseconds, exit do_lcd() case 2: // do state 2 stuff state=3; after(M, do_lcd); return; // set next state, use after to delay execution by M milliseconds, exit do_lcd() } }

The busy waiting version of do_lcd() was: do_lcd() { // do state 1 stuff delay(N); // do state 2 stuff delay(M); } }

The do_lcd() function is ugly, but it is working without busy waiting. If all Arduino users are annoyed by event handling uglyness, they are ready for phase 3 of the RTOS brainwash program:

Migration step 3: Show that multi-threading does look better

Multi threading is dangerous. The worst thing you can do with busy waiting is to freeze everything up. The worst thing in multi threading is that you literally tear apart your machine because thread 1 says "go left" and thread 2 says "go right" as fast as time-division multiplexing can do. But, if you have multi-threading and a multi-threaded version of delay(), the do_lcd() function will no longer look ugly, and you are still out of busy waiting land:

do_lcd() { // do state 1 stuff delay(N); // the multi-thread version of delay() will switch to another thread or will wait in the operating system // do state 2 stuff delay(M); // the multi-thread version of delay() will switch to another thread or will wait in the operating system } }

I bet, everybody will be happy about a keyboard() lib. And they will learn that "fu**ing" event crap, because using a lib is better then writing a lib. And everybody who uses keyboard() will use after() and every(). And now the future is open: If you can live with the event programming uglyness, you never have to go to multi-threading land. Maybe you remember the X windows system, the GUI for UNIX and Linux? Well this was all done by using event programming, and it was done a long time before multi-threading was invented.

Well, I don't know exactly what you mean by this, do you condone one way or the other, but I for sure am not a software architect and there for will not write my own state machine, I expect this to be handled by the framework so I can focus on my logic level. I expect to write a couple of lines of code to get a couple of lines of logic nothing fancy nothing hard.

Yeah, I think we need to stop thinking of the implementation, and start thinking about what the API should look like if it's going to be usable by "The Arduino Community."

timer mytimer; void timerSet(mytimer, timeInms) void timerSet(mytimer, hours, int minutes, int seconds) boolean timerExpired(mytimer) long timerTimeLeft(mytimer)

alarmClock myclock alarmClockSet() alarmSet(endTime)

In this case "timers" implement a delay for some duration, while alarmclocks implement checking whether some absolute time has ocurred.

If you want to cater for a hobbyist like myself with a non-software background, then it's essential to provide good clear documentation.

I for one will alway choose a library of any complexity that has good clear documentation, even at the expense of overlooking a possible more suitable one where I'm expected to wade through the code to deduce its workings.

My interest in scheduling is because I'm reading a few sensors with an Arduino.. I'm using Dr Liu's 'phi_prompt' menu library to select and display the values on an LCD. Unfortunately the phi_prompt library sits in loops waiting for user input, during which the sensors aren't read, so I'm looking for a way round the problem and suspect the way forward may be some sort of scheduling system. I'll look at NilRTOS (thanks 'fatlib16'!), but I've not seen a big fat PDF file describing it, so I may not get far!

Jim

westfw: Yeah, I think we need to stop thinking of the implementation, and start thinking about what the API should look like if it's going to be usable by "The Arduino Community."

timer mytimer; void timerSet(mytimer, timeInms) void timerSet(mytimer, hours, int minutes, int seconds) boolean timerExpired(mytimer) long timerTimeLeft(mytimer)

alarmClock myclock alarmClockSet() alarmSet(endTime)

In this case "timers" implement a delay for some duration, while alarmclocks implement checking whether some absolute time has ocurred.

How about simply "sleep(x);" like ...

code here; /* Last line to run before sleeping / sleep(100); / Thread goes to sleep here / code here; / Thread wakes up here after delay */

sleep(); causes THIS thread to release the CPU back to the RTOS. Other threads can run while this thread is sleeping. When the Timer Expires then the Interrupt allows the RTOS to "wake up" the sleeping thread.

sleep(); causes THIS thread to release the CPU back to the RTOS.

No, no! If we had an RTOS with threads we can just use delay() (naming conflicts aside.) I want to know what APIs we can use FROM NOTHING, because I think basic RTOS concepts are too difficult.

westfw:

sleep(); causes THIS thread to release the CPU back to the RTOS.

No, no! If we had an RTOS with threads we can just use delay() (naming conflicts aside.) I want to know what APIs we can use FROM NOTHING, because I think basic RTOS concepts are too difficult.

OK, Use the word "delay(x)" . It does not matter what you name. Although, many RTOS use sleep() as it more accurate description of what actually occurring. It was the concept that: "THREADS SHOULD NOT BLOCK". This is one difference between multi-threaded vs. single thread. And is the best way to utilize CPU resources.

Also, if you want to use a Single Thread in RTOS then you can still do that. It is not required that you make you App multi-threaded. Multi-Threading Is an option that is available if you have the skill set to use it.

Much like using the more advanced "Interrupt Driven" Threads we can use now.

Happy New Year everyone!

I would like to weigh into the debate on this one... with a resounding YES we need an RTOS, however I recognize that the majority of users will not know how to implement it due to a procedural mindset and thats not a bad thing but for the rest of us, the ability to have tasks running and doing stuff is perfectly natural.

To date the CHibios port looks great, FreeRTOS might have too much over head, but one thing we will need in it from day 0 is a fully working Ethernet Library and all the existing libraries will need to be task/thread safe.

I'm old school, started coding back in 1976 but I also can readily adapt to change when change is good, I do a lot of work in message queuing systems at present and write a lot of C++ code. To me kicking off jobs across 100's of queues so tasks are decoupled and running in various states of parallelism is perfectly logical, but for the majority of programmers they just dont get it... so the RTOS would need to be so well integrated you dont have to think about it. Then when the skill level of the coder rises he/she will start to use tasks and think in terms of lots of stuff happening at once.

I have had a lot of exposure to hardware and software for just about every industry and spent many years doing battle in the embedded space... I think for the Arduino to jump into the commercial world it needs an RTOS. and it is a commercial product if you look at just how much stuff is available for it... my other bug bear is the mounting holes... if you design an board, please make the holes line up! (Rant over).

Sid