Arduino Forum

Development => Suggestions for the Arduino Project => Topic started by: fat16lib on Jan 13, 2013, 06:10 pm

Title: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 13, 2013, 06:10 pm
Do Arduino users even want multitasking?  I would be interested in any feedback.

Recently there have been many proposals for an "Arduino Scheduler".

One of the latest, SCoop, even claims to be an "Android Scheduler".  I find this claim to be amazing if the Linux scheduler with cgroups has been implemented on Arduino.

Each proposal claims to have a "simple powerful multitasking solution". 

It's not clear what problem these schedulers intend to solve.  Some proposals claim to be for "real time operating systems".  Other proposals are for schedulers with weak real-time capabilities.

So what is the requirement for an Arduino scheduler?  Only Arduino users can answer this.

You don't need a scheduler unless you want multitasking.  So do you, as a user, want multitasking?  I say "want" not "need" since you can always invent your own solution starting with "bare metal".

If you want multitasking do you need soft real-time?  Soft real-time optimizes resource use hoping to improve quality of service in systems like Android, Mac, and Windows.

Do you need hard real-time?  Hard real-time uses less than 100% of computing resources but guarantee response within strict time constraints.

Performance is not the only reason to use a given scheduler/RTOS.  A RTOS can help simplify a system's architecture by allowing a system to be decomposed into independent tasks and using OS services such as message queues, mutexes, semaphores, event flags, etc. to communicate & synchronize, the system.

A common misconception is that a preemptive RTOS adds an unacceptable amount of overhead.  In fact, a preemptive RTOS will only require between 1 and 4% of CPU time in exchange for valuable services.

A cooperative scheduler is often very inefficient since many unnecessary context switches occur as the result of yield calls.

So what is your opinion about Arduino multitasking and schedulers?
Title: Re: Does Arduino need a real-time scheduler?
Post by: retrolefty on Jan 13, 2013, 06:23 pm
Quote
So what is your opinion about Arduino multitasking and schedulers?


As a pure hobbyist with mostly a hardware background I have no idea if any of my typical or future projects would benefit from having access to a multitasking and scheduler environment to work inside of.

Most of the beginners questions about needing or wanting multitasking seems to me to be more just a mental block on their part because of not understanding basic C/C++ program structure ability and that one just normally needs to address avoiding blocking functions so that their main loop function can handle all the independent tasks they wish to accomplish in their sketch in a timely matter. Then add pretty simple to understand user and pinchange interrupt capabilities and I just haven't seen the need to have or learn a more complex environment to work with.

So I guess I would certainly be interested from a learning perspective but I suspect if it gets too complex to understand or actually implement, I would probably never get around to actually trying it, not unlike the gazillion arduino libraries I've downloaded, looked at quickly and said "nope, it's over my head".  ;)

Lefty
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 13, 2013, 07:23 pm
retrolefty,

I suspect you're in the majority and few will use true RTOS features.

Reality is a good thing, I often expect Arduino users to do what young students do here in introductory embedded systems courses.  Edward A. Lee recently modernized the introductory EECS 149 course. 

Here are some example projects. This video starts with a face tracking project.  A camera on a quad-copter tracks the student's face, stays at the same level about four feet away.

http://www.youtube.com/watch?v=VqLlvrMsWc0 (http://www.youtube.com/watch?v=VqLlvrMsWc0)

The lectures for this course are on youtube http://www.youtube.com/playlist?list=PL62BE418C34C3B2BD (http://www.youtube.com/playlist?list=PL62BE418C34C3B2BD).  Slides are here http://chess.eecs.berkeley.edu/eecs149/lectures/index.html (http://chess.eecs.berkeley.edu/eecs149/lectures/index.html)

The book for this course is somewhat theoretical but is a good reference and free http://leeseshia.org/releases/LeeSeshia_DigitalV1_07.pdf (http://leeseshia.org/releases/LeeSeshia_DigitalV1_07.pdf).

Here is another version student projects videos:
https://www.youtube.com/channel/UCS7MdBus8jygYZy76rIO0pw?feature=watch (https://www.youtube.com/channel/UCS7MdBus8jygYZy76rIO0pw?feature=watch)



Title: Re: Does Arduino need a real-time scheduler?
Post by: westfw on Jan 14, 2013, 11:49 am
Quote
So what is the requirement for an Arduino scheduler?  Only Arduino users can answer this.

I doubt that.  I would guess that at least 80% of professional software engineers working in embedded systems would do a poor job in specifying "requirements for a scheduler" for their platform.  50% probably confuse "real time" with "fast."  Arduino users are nearly by definition less aware, and aren't interested in being aware of those issues.

Their answer is probably "I want to be able to do several things at one time."  That's probably what their answer should be.  That's the way most desktop users think of things as well.
If you can get some quantitative answers for the value of "several", and the limitations (timewise) of "at once", you'd be doing really well.  Consider:
Code: [Select]
task foo1 {at noon digitalWrite(1, HIGH);}
task foo2 {at noon digitalWrite(2, HIGH);}
  :
task fooN {at noon digitalWrite(N, HIGH);}

What's an appropriate value for N, and how many milliseconds after noon do all the N pins need to be high?
Is it more important for that number of milliseconds to be small, or to always be the same?
THIS kind of question you might be able to get answers for.

Quote
a preemptive RTOS will only require between 1 and 4% of CPU time in exchange for valuable services.
*will* required, or *could* require?  I find that number hard to believe, depending on what it is measuring.  Although... CPU time is probably not the bottleneck resource, so it probably doesn't matter.
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 14, 2013, 04:00 pm
westfw,

Quote

Their answer is probably "I want to be able to do several things at one time."  That's probably what their answer should be.  That's the way most desktop users think of things as well.
If you can get some quantitative answers for the value of "several", and the limitations (timewise) of "at once", you'd be doing really well.  Consider:

Your almost certainly right.

I have experience with two types of users, research physicists, and EECS students from UC Berkeley.  Both groups are comfortable using a RTOS.  Older EEs not so much.

I am a PhD physicist and I have done architecture and design of control systems for many large experiments.  We started using RTOSs about 40 years ago.

I did some work on LHC, the big experiment at CERN looking for the Higgs Boson.  CERN has used LynxOS in control systems for over twenty years.

Some of my colleagues left the lab to develop VxWorks.  NASA JPL uses VxWorks in all Mars rovers.

I find it hard to understand the resistance to use of RTOSs.

Maybe a tutorial with more realistic example application would help Arduino users understand the value of RTOSs.  Cortex M clearly was designed for use of an RTOS.

On the other hand, unless you understand things like reading an ADC at relative slow rates, like 100 Hz, requires a time jitter on the order of one microsecond if you want low SNR in the signal.  The SNR of an ideal 10-bit ADC is about 62 dB.  At 100 Hz, 4 microseconds of jitter in the reading time reduces the SNR to about 52 dB.  A coop scheduler just won't schedule a thread with low jitter and reading a sensor in an OS thread is easier than setting up a timer driven ISR. 

Quote

*will* required, or *could* require?  I find that number hard to believe, depending on what it is measuring.  Although... CPU time is probably not the bottleneck resource, so it probably doesn't matter.


The measure generally means CPU time.  A RTOS has no extra overhead unless you call a OS function or an event causes a context switch.  You don't do a context switch for every interrupt and many fast interrupts can be handled just like the bare metal approach without any OS overhead. For example a serial driver puts bytes in a queue just like the Arduino drivers.

On a chip like a 72 MHz STM32, a context switch with ChibiOS costs just over one microsecond.  A well designed application should not have more than a few thousand context switches per second so the overhead will be a few percent. 

A preemptive RTOS responds to an important high priority event with the handler thread running in one microsecond but with a coop scheduler, who knows.
Title: Re: Does Arduino need a real-time scheduler?
Post by: pito on Jan 14, 2013, 08:09 pm
Quote
I find it hard to understand the resistance to use of RTOSs.

That is not about the "resistance" but the need. Most arduino users do not work for LHC or NASA and their designs work just fine with a superloop ;)
With more demanding applications they will certainly consider an rtos..

Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 14, 2013, 09:43 pm
Quote

That is not about the "resistance" but the need. Most arduino users do not work for LHC or NASA and their designs work just fine with a superloop


pito,

Given a choice of a supperloop or two simple tasks, scientists who write device code choose the simpler more reliable task model.  A RTOS provides better partitioning of an application even in simple cases.

Old EEs make the funny spaghetti supperloop that mixes timing and code for two distinct operations.

I think experimental physics groups adopt things like RTOSs more readily because the learning overhead is shared.  Members help each other, the first person to learn the system helps the next person.

A RTOS is not like learning a new programming language.  It requires a different architecture for embedded systems.  That's why modern embedded systems text books don't emphasize details of the OS.  These books cover things like why preemption is required for rate monotonic scheduling and why this is important for reliable systems.


Title: Re: Does Arduino need a real-time scheduler?
Post by: westfw on Jan 15, 2013, 12:06 am
Quote
I find it hard to understand the resistance to use of RTOSs.

Well, there's one set of people who doesn't understand what an RTOS is, what it would give them, or how they'd choose between multiple options.  They have enough problems figuring out how to divide a program into functions, much less into concurrent "tasks."

There's another set that understands, but is worried about the complexities that you know or suspect comes with it.  Perhaps they've been burnt by a negative experience with an existing RTOS.  Or they're worried that they don't want to increase latency to get certainty.  Or they're just comfortable, given the size of Arduino, that they can get along without it.  Case in point:
Quote
CERN has used LynxOS in control systems for over twenty years.
Some of my colleagues left the lab to develop VxWorks.

So given 20 years of experience with an RTOS, your coworkers were so frustrated with it that they went to work on a different RTOS ?  :-)

Quote
A RTOS is not like learning a new programming language.

The hell it isn't.  Especially if the product is already intentionally blurring the lines between "language", "library", and "run time environment."
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 15, 2013, 01:27 am
westfw,
Quote

So given 20 years of experience with an RTOS, your coworkers were so frustrated with it that they went to work on a different RTOS ?  :-)

Wrong!  They didn't work on a different system, they commercialized the open Berkeley system as VxWorks and NASA started using it.  CERN picked LynxOS in Europe at about the same time.   CERN and LBNL are happy with their systems.

VxWorks now has over a billion copies in products.  Several other commercial RTOSs also claim over a billion copies in products.

RTOSs aren't that different in basic functions.  Almost all commercial RTOSs have a fixed priority preemptive scheduler.  They have similar synchronization/communication primitives. I find it easy, almost mechanical to convert a program from one to another.

You can't choose an RTOS because you don't understand the associated theory.  That's what engineers learn in courses like UC Berkeley's EECS 149.

You comments are valuable.  I get the message that you will never use a RTOS. 

At least you are not like the old engineer I knew who programmed a ROM for one of the first micro-controllers by filling in squares on a engineering pad and then entering them in switches in his home made programmer.  I never got him to use an assembler.  That was around 1971 or 1972 and it might have been a 4-bit 4004.  We quickly moved to the 8008 but left the old EE with his pad behind.

Title: Re: Does Arduino need a real-time scheduler?
Post by: pito on Jan 15, 2013, 10:29 am
What kind of rtos did you use with 8008?
Title: Re: Does Arduino need a real-time scheduler?
Post by: PaulS on Jan 15, 2013, 12:45 pm
Quote
Given a choice of a supperloop or two simple tasks

I know which a Hobbit would choose.
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 15, 2013, 02:26 pm
PaulS,

Sorry, I depend too much on seeing red for spelling errors but it doesn't work for superloop/supperloop.

pito,

We did some prototyping on the 8008 and decided it wasn't flexible enough.  We were in contact with MOS Technology which was formed in 1972 so we moved to the 6502.

We built a little kernel and attached lots of RAM since there was no flash then.  We didn't name the system.

Of course Apple used the same chip for the Apple I in 1976 and Apple II in 1977.

We attached these systems to a serial port on a terminal server and downloaded programs from a CDC 6600 supercomputer.  We programmed in PL/M, not assembler.  The cross compiler ran on the 6600.

One reason I find Arduino interesting is that its technology is so much like what I was doing 40 years ago.

By 1980 commercial kernels like VRTX were available for chips like the 68000 and we moved beyond Arduino style systems.
Title: Re: Does Arduino need a real-time scheduler?
Post by: ddmcf on Jan 15, 2013, 04:07 pm
I'm probably not a typical Arduino enthusiast, but I almost immediately looked for a scheduler for my first project. Maybe out of preference because I know I could do my project with a super loop, and even an RTOS, but all I really needed was a scheduler with inter-process communications. I say "need" because that is my preference for this particular project.

In my retirement I'm building a smart toy for my granddaughter that will have up to 8 I/O's that need to be handled asynchronously in "real time". I could not find a simple scheduler (and I did ask on the forum as well) so I adapted the Quantum Leaps code and built what I call an asynchronous (non pre-emptive) device framework on top of it. It is more than adequate for my project needs, and is scalable for future projects. I admit that I had fun building the framework, but I would have easily used something already available.

Because I feel strongly that a scheduler is a good tool for solving specific problems, and I was unable to find one, I wrote about how I adapted QF to make it easier for others to do the same:

http://wizmoz.blogspot.com/2012/12/quantum-leaps-arduino-state-machine.html (http://wizmoz.blogspot.com/2012/12/quantum-leaps-arduino-state-machine.html)

There are a wide range of user skills and project complexities in this forum. The hobbyist that needs to blink some LED's isn't going to need a full blown scheduler or an RTOS, and someone writing code for particle collisions probably doesn't want one; but there are some of us here in the middle that would benefit from such a scheduler, whether a kid toy project or something more demanding. Absolutely, yes.
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 15, 2013, 04:41 pm
ddmcf,

I agree, the Quantum Leaps state machine framework is an excellent type of scheduler for applications like your project.

What kind of feedback have you received?  Is your tutorial sufficient to get people started?

I am curious how Arduino users react to state machines. 

Students are introduced to finite state machines early in introductory embedded systems courses.
Title: Re: Does Arduino need a real-time scheduler?
Post by: westfw on Jan 15, 2013, 06:09 pm
Quote
I get the message that you will never use a RTOS. 

I wouldn't go that far.  I can't see filling up the memory of a MEGA or DUE without more structure (in the form of SOME sort of OS.)  I doubt whether I'd ever need the "real time" aspects, but that seems to be what is available; I can't see writing my own, especially when the existing rtoses are getting pretty favorable reviews.

Most of my professional career was spent programming under a proprietary, non-preemptive, not real-time OS.  OTOH, that company's experiments with real RT kernels was less than spectacularly successful.   When your uart ISR starts sending messages instead of just reading the chip, something has "jumped the shark."
Title: Re: Does Arduino need a real-time scheduler?
Post by: ddmcf on Jan 15, 2013, 06:59 pm
fat16lib,

I've gotten some good feedback in the form of "thanks for writing this up", but no specific additional questions on the QF hack. I've been pleased with the traffic to the specific blog post on state machines, it seems to be gaining momentum over the last month, perhaps in light of recent talk about schedulers like this.

Thanks for this post, I believe it builds awareness of some of the tools that are available for solving various control problems.



Title: Re: Does Arduino need a real-time scheduler?
Post by: drjiohnsmith on Jan 16, 2013, 08:04 pm
to answer the question
   yes it dosn't

depends what your doing with the thing.

would be nice if a simple rtos was available,

as for the comment back a while about the good old 8008,
   the work on the apollo computer used a rtos / interupt / schedular system,
     its what saved the 11 mission when buzz left the  return to orbit  radar on as well as the landing radar.

http://ed-thelen.org/comp-hist/vs-mit-apollo-guidance.html

have fun
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Jan 17, 2013, 04:35 pm
Would an extremely small preemptive RTOS appeal to Arduino users?

I have been playing with an experimental RTOS written by Giovanni Di Sirio, the author of ChibiOS/RT. 

His goal is to build the smallest possible kernel for tiny chips.  Giovanni calls the system Nil RTOS since the goal is a zero size kernel.

Nil RTOS has only the most fundamental functionality.

A preemptive fixed priority scheduler.

Counting semaphores that can signal from a thread or ISR.

Sleep until a specified time and sleep for a specified period.

Here is an example sketch with a total size under 2KB  on an Uno:
Code: [Select]

// Connect a scope to pin 13.
// Measure difference in time between first pulse with no context switch
// and second pulse started in thread 2 and ended in thread 1.
// Difference should be about 10 usec on a 16 MHz 328 Arduino.
#include <NilRTOS.h>

const uint8_t LED_PIN = 13;

// Semaphore used to trigger a context switch.
Semaphore sem = {0};
//------------------------------------------------------------------------------
/*
* Thread 1 - high priority thread to set pin low.
*/
NIL_WORKING_AREA(waThread1, 128);
NIL_THREAD(Thread1, arg) {

  while (TRUE) {
    // wait for semaphore signal
    nilSemWait(&sem);
    // set pin low
    digitalWrite(LED_PIN, LOW);
  }
}
//------------------------------------------------------------------------------
/*
* Thread 2 - lower priority thread to toggle LED and trigger thread 1.
*/
NIL_WORKING_AREA(waThread2, 128);
NIL_THREAD(Thread2, arg) {

  pinMode(LED_PIN, OUTPUT);
  while (TRUE) {
    // first pulse to get time with no context switch
    digitalWrite(LED_PIN, HIGH);
    digitalWrite(LED_PIN, LOW);
    // start second pulse
    digitalWrite(LED_PIN, HIGH);
    // trigger context switch for task that ends pulse
    nilSemSignal(&sem);
    // sleep until next tick (1024 microseconds tick on Arduino)
    nilThdSleep(1);
  }
}
//------------------------------------------------------------------------------
/*
* Threads static table, one entry per thread. Thread priority is determined
* by position in table.
*/
NIL_THREADS_TABLE_BEGIN()
NIL_THREADS_TABLE_ENTRY("thread1", Thread1, NULL, waThread1, sizeof(waThread1))
NIL_THREADS_TABLE_ENTRY("thread2", Thread2, NULL, waThread2, sizeof(waThread2))
NIL_THREADS_TABLE_END()
//------------------------------------------------------------------------------
void setup() {
  // Start nil.
  nilBegin();
}
//------------------------------------------------------------------------------
void loop() {
  // Not used.
}


I wrote this sketch to determine the performance of Nil RTOS.  I was amazed to find how fast it is.  The time to signal a semaphore, do a contex switch and take the semaphore is only about 12 microseconds on an Uno.
Title: Re: Does Arduino need a real-time scheduler?
Post by: dbasberg on Feb 02, 2013, 02:36 am
I have read all of this thread and found it very interesting.  I worked on powertrain controllers as a consultant for brands x and y.  They both had a mechanical engineering mind set in the early days and put any old EE on the coding for the controllers.  Glad to say that is no longer the case.  Some went kicking and screaming from absolute assembly to relocatable assembly.  Then the same with going to "C".  And, again for a RTOS.  At this time they are doing model based control algorithms and auto code generation with an RTOS.  There were safety concerns about allowing higher level interrupts interrupt lower level interrupts.  It was believed that circumstances might arise that could not be reliably predicted.  I disagreed, but I was probably wrong.  A friend working at Wind River was involved in the code for the Mars rovers.  You may recall the first rover froze up.  The software locked up.  The problem was task scheduling in the very complex multitasking RTOS.  Luckily, or cleverly, they had some code in the system that recognized how much trouble it was in and went into a mode to accept new code by telemetry.  So, by the time the second rover landed, they had the fix.  I am an old engineer 65+ and age is not the problem, mind set is.  You are either open to new ideas, or not.  I have a hobby farm and I am retired, so my current project is to use a Raspberry Pi(RPI)... (google) and an attached I/O board with an ATmega328p on it.  The ATmega is much more powerful that the first chips we ran the engine with which was (1K RAM, 16K ROM, and 2MHz).  We used a 10msec interrupt to read the tone wheels for RPM and schedule background tasks.  It was a rudimentary O/S for scheduling and accurate sensor reading.  The building security/monitoring/controlling I will do for three out buildings on the property will use a similar rudimentary O/S.  The ATmega is a slave to the RPI on a 115K serial UART channel.  The RPI is a powerful processor with 512K RAM and SD flash ROM (8Gig) with netork connector and two USB connectors.  I will run WiFi on one of the USB ports. The RPI runs Debian Linux and can do an Apache server, if you wish.  The cost is very low.  The RPI is $35 and the Gert I/O board is $48.  I was working on computerized test equipment at Bell Labs when the Intel 4004 and 8008 came out.  We were doing 148 pin circuit board testers and wanted to go to a processor per pin, but the 8008 and 4004 were not fast enough or powerful enough.  We stayed with DEC and Data General minicomputers.  Anyway, this thread was very interesting and I think that the power of the Arduino boards does warrant the use of a periodic interrupt and simple task scheduler for many applications.  I have used the 'MicroC/OS-II' real-time kernel by Jean J. Labrosse, which has been ported to many uPs.  It does a fine job and allows the user to pick features and leave out features to arrive at the proper size and power.  Worked well for me on my greenhouse controller, which reports over the internet.  I have seen another similar featured RTOS called 'freeRTOS'...(google) that costs nothing.  I think it would be useful for some to look at these.  Have fun :D.
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Feb 02, 2013, 12:03 pm
Much was made of the Mars rover bug but it was just that, a bug.  Like all bugs, it shouldn't have happened since the proper design for avoiding "priority inversion" was well known since the early 1970s.  The Mars rover problem happened in 1997.
Quote

When created, a VxWorks mutex object accepts a boolean parameter that indicates whether priority inheritance should be performed by the mutex. The mutex in question had been initialized with the parameter off; had it been on, the low-priority meteorological thread would have inherited the priority of the high-priority data bus thread blocked on it while it held the mutex, causing it be scheduled with higher priority than the medium-priority communications task, thus preventing the priority inversion. Once diagnosed, it was clear to the JPL engineers that using priority inheritance would prevent the resets they were seeing.


I did become part of the fear factor of using a RTOS.  Like this kind of misinformation - "an RTOS shouldn't be used on an Uno since the overhead is too high".

Here is a simple case study for an example I am developing.  The problem is to read data from analog pins at regular intervals and write it to an SD card.

The simple solution is a loop like this:

1. Wait till start of period.

2. Read data.

3. Write data to SD.

4. Repeat.

A problem occurs when the period between points is less than about 100 milliseconds.  SD cards can have occasional latencies of over 100 milliseconds so data overruns occur.

A possible solution is to use an RTOS with two threads.  Can the Uno support the extra overhead?

The answer is that the RTOS solution is far more efficient than the above loop.  Here's why.

The RTOS solution has two threads.

The analog read thread runs at high priority and is a loop like this:

1. Wait till start of period.

2. Read data.

3. Write data to a FIFO buffer.

4. Repeat.

The SD write thread is a loop that runs at lower priority.

1. Wait for data in the FIFO.

2. Write data to SD

3. Repeat.

The two thread solution is more efficient than the first single loop solution.  CPU time is recovered when the SD is busy and the higher priority thread is scheduled.

Now comes the real payoff.  The Arduino analogRead() take about 115 microseconds.  Almost all of this is in a busy loop waiting for the ADC conversion.

I wrote an RTOS based replacement for analogRead() that is transparent to users but sleeps during the ADC conversion.  This saves over 90 microseconds of CPU time per read after factoring in a context switch.

The result is that the RTOS version can log more than twice as fast and doesn't suffer data overruns.  The simple loop version has the high overhead of busy loops that the RTOS avoids.

I have ported three RTOSs to Arduino http://code.google.com/p/rtoslibs/ (http://code.google.com/p/rtoslibs/).

My favorite for Uno is NilRTOS.  Its author, Giovanni Di Sirio, says it's "Smaller than ChibiOS/RT, so small it's almost nil."
Title: Re: Does Arduino need a real-time scheduler?
Post by: bboyes on Feb 05, 2013, 06:19 pm
Interesting thread. I have been working on commercial hard- and soft- real-time control systems for a number of years. Since those terms are commonly misused and misunderstood here is an example of a hard-time system we did a few years ago: a mud-pump controller driving two pistons with 2-meter stroke, controlled with about 500 HP of hydraulic pumps (at 10,000 psi, the manifold pipe is something like 200 mm in dia), through a servovalve for each piston. Each piston moves by a polynomial equation and their outputs are summed through a check valve so that the flow is constant, and adjustable over a wide range. LVDTs with 2-meter stroke monitor each piston. Smooth startup and shutdown and several considerations of fail-safe behavior were used. We achieved this on an 8051 with external ADCs and DACs, and interrupt handlers (two levels, pre-emptive) with re-entrant libraries. We almost ran out of code space. In our case the time interval was not tiny, but we had hard completion deadlines. The piston profile came out of a lookup table (actually 1/4 of the entire profile, mirrored and phase-shifted as needed), the actual piston position was read, and that was all fed into a firmware PID routine which then calculated the next value for the servovalves. That had to be completed before the timer tick to update the valves. That timer got faster as the flow rate was increased. Then in the background was interrupt-driven serial I/O for the machine interface to control the mud pump and to monitor critical performance values (such as the current error value vs target position). This I/O had to be safely pre-empted by the piston routines with due consideration of atomicity of variables which might be in the midst of being updated by the control interface. In the end it all worked well, we delivered complete documentation and source code, and I have not heard of any problems. I went to Japan to help install and tune the system and then our part of the project was complete.

Where execution deadlines got tight I would set and clear some spare I/O bits and watch them on an oscilloscope or logic analyzer, something like High when active in a critical routine and Low when in safe extra time margin. As the flow rate ramped up you could see the bit transitions coming closer together. If they ever collided that would be potentially catastrophic since the control loop could not correctly function. We set the scope to trigger on the smallest safe interval and left it running overnight (without driving the actual pumps - simulated input). There was also simple instrumentation in the code to log deadline violations.

The project engineer was a delight to work with: a very experienced, practical guy. One example: we had to monitor the end-of travel limit clearance of the pistons (which weighed over 1000 kg as I recall) since we didn't ever want to drive them into their stops. No one knew what would happen if that occurred. Safe clearance was something like 5-10 mm, and it could not be observed easily by eye while running. His brilliant idea was to use an empty aluminum soft drink can in the gap and measure the crushed thickness. Worked great and no risk of harm to the pistons.

In this project a simple RTOS might have been a huge timesaver. But we also had to have timer interrupts and serial I/O interrupts and I am not aware of how easily RTOSes can fit with those. If it is possible to weave RTOS features into your own needed I/O hardware support, that could be helpful but also could get a bit complex.

I am using Teensy++2 on a project now and just got the ARM Teensy 3, and there is enough code and data space on these new Arduino devices. My concern would be: is the RTOS granular so we only need to use what fits our case, and can it co-exist with I/O device interrupt handlers?

Thanks
Title: Re: Does Arduino need a real-time scheduler?
Post by: fat16lib on Feb 05, 2013, 09:35 pm
Quote

In this project a simple RTOS might have been a huge timesaver. But we also had to have timer interrupts and serial I/O interrupts and I am not aware of how easily RTOSes can fit with those. If it is possible to weave RTOS features into your own needed I/O hardware support, that could be helpful but also could get a bit complex.

Quote

My concern would be: is the RTOS granular so we only need to use what fits our case, and can it co-exist with I/O device interrupt handlers?

A good RTOS allows independent interrupt handlers.  The three systems I ported to AVR/ARM Arduinos can be used as simple kernels with all I/O from the Arduino core or custom libraries.

I am starting to write optional I/O libraries that are integrated with the RTOS.  Many existing Arduino I/O functions just do busy loops so putting a thread to sleep and waking it with an I/O done interrupt is more efficient.

If you need a highly optimized ISRs, systems like ChibiOS are designed to allow your ISR to run with no interference from the RTOS.

On ARM this works very well with priority interrupts.  ChibiOS is designed to allow "Fast interrupts".
Quote

On some architectures ChibiOS supports a special class of "Fast Interrupts", such interrupt sources have a higher hardware priority than the kernel so it is not possible to invoke system APIs from there.

The invocation of any API is forbidden here because fast interrupt handlers can preempt the kernel even within its critical zones in order to minimize latency.


The only hardware resource I uses is access to a timer interrupt for the system tick. 

I piggyback on sysTick on ARM since both Teensy 3.0 and Due will have a hook.  So no new hardware is used on ARM.

On AVR I am using the timer 0 compare A interrupt.  I did this so no Arduino core code needs to be changed.  I will release an optional patch to piggyback on the timer 0 overflow interrupt.  This is already used for delay(), millis(), and micros().  Then not new hardware will be used on AVR.
Title: Re: Does Arduino need a real-time scheduler?
Post by: Se3ker on Dec 03, 2013, 10:35 am
I first started with Netduino, then switched to Arduino because of more hardware and components and also cheaper, I'm now fiddling with some RF components and I really really need Threads, I don't know exactly what threads are in .NET and what is the importance of RTOS. But I need precision timing, and execution from a couple of wirelessly enabled arduino units. Think of it as a wireless sequential light system, I messed with millis(), timers and lots of libraries, all are unpredictable and hang. Or just drift apart as time goes by in execution. I dunno maybe im doing something wrong, but when I used Netduino I could do this preety easily.

Anyway its explained like this:

Quote

The .NET Micro Framework CLR has only one thread of execution and owns all of the memory in the system. During normal operation, the CLR iterates in the interpreter loop and schedules managed threads using a round-robin algorithm, according to the priority of the threads involved. Each managed thread gets a 20-millisecond (ms) time quantum during which the runtime executes intermediate language (IL) code that belongs to the stack of the managed thread being serviced. When the managed thread goes to sleep or waits for a synchronization primitive, such as a monitor that cannot be acquired or an event that is not signaled, the CLR puts the thread in the waiting queue and tries to schedule another thread. In between scheduling managed threads, the CLR checks for any hardware events that might have been raised at the native device driver level. If an event occurred, the CLR tries to dispatch the event to the managed thread that requested it. A hardware event is associated with some kind of input/output (I/O) event, whether it be an interrupt-based GPIO event or a serial USART, USB, or I2C event.

The runtime interpreter loop starts by checking to discover whether a thread is ready for a time slice. If a thread is ready, the CLR schedules the thread for processing and tests to determine whether any hardware interrupt events require processing. If there are, the CLR processes the hardware interrupt event and moves the thread that handles the interrupt into the queue of threads that are ready to be processed. If no hardware interrupts have occurred, the CLR again tests to see whether a thread is ready for a time slice.

If the CLR determines that there are no threads that are ready to be processed, it sets a timeout timer and goes to sleep in order to conserve power. When the timeout timer triggers or an interrupt occurs, it wakes the CLR back up. At that time, the CLR determines whether an interrupt or the timer woke it up. If it was an interrupt, the CLR determines if processing the interrupt is required. If not, it sets the timeout timer and goes back to sleep. If it is the timeout timer that wakes the CLR, the CLR executes any pending completions and checks again for interrupts. If there are waiting interrupts, it goes through the normal interrupt processing procedures. If there are no waiting interrupts, the CLR sets its timeout timer and goes back to sleep.


Basic example with Netduino
Code: [Select]

namespace NetduinoApplication1
{
    public class Program
    {

        static Thread myThread = new Thread(new ThreadStart(myFunc));
       
        public static void Main()
        {
            myThread.Start();

            //suspend at some point, wait for event
            //myThread.Suspend();

            //resume at some point, wait for event
            //myThread.Resume();

            Thread.Sleep(Timeout.Infinite);
        }

        public static void myFunc() {
            // do some stuff
        }

    }
}
Title: Re: Does Arduino need a real-time scheduler?
Post by: AndreAdrian on Dec 15, 2013, 09:51 pm
My opinion: Arduino needs a better delay() function  :smiley-mr-green:

The question RTOS or no RTOS does just heat up the discussion, but does not help the migration. This is my approach:
I want a better delay() function - this is a feeling. If I introspect this feeling I recognize
- it is a waste of energy for me if delay() is just doing busy waiting.
- I want/I need to do several things in parallel, but delay() does not allow me to do it.
- I do not want to throw everything away because of a bigbang RTOS solution, I want to go step by step from here to there.

As a seasoned software engineer I have seen again and again that migration is the key. Here are my suggestions:

Migration step 1: Introduce event handling
--------------------------------------------
The Arduino "core library" has the loop() function. An event handling Arduino program will have one only statement in the loop() function, the do_event() function. The do_event() function has to look after the events and has to service the events.
A useful vehicle to show that event handling is useful should be a keyboard scan library that provides its services within an event handling framework. The typical pocket calculator keyboard has arranged the keys between row and column lines. A 12 button keyboard needs 3+4=7 scan lines. The keyboard scan function is not trivial. Everybody expects to get:
- key debouncing
- key repeat (repetition)
- 2-keys-rollover (Rapid typists may sometimes inadvertently press a key before releasing the previous one)
- no electrical shortcircuit if two keys are pressed simultaneously
To implement these functions, a timer is needed. The keyboard lib should provide a timeout functionality to the user. The function set of the keyboard lib is:
button_register(): combine an event (key press) to a function (eventhandler)
after(): after N milliseconds call a function
every(): every N milliseconds call a function
do_event(): check if there is a pending event and execute the eventhandler function. Needs to be called again and again within loop().

A simple Arduino program that fits nicely the event paradigma is a math trainer program. The Arduino displays a math question like "1 + 2 = ?" and the user should enter the correct answer within a time limit. In an event program there will be a keyboard event for every digit and a timeout event. If the user presses the digit "3", the button-3-event function will say "correct answer". The other button-event functions will say "wrong answer". And the timeout event-function will say "timeout".

Migration step 2: Create a event-driven LiquidCrystal lib
----------------------------------------------------------
Everybody is using LiquidCrystal. But LiquidCrystal uses delay() and busy waiting is the nemesis to all real time programming. Thanks to the keyboard lib above, we have one-shot timer. And an event driven LCD output function does look like this:
do_lcd() {
switch (state) {
case 1: // do state 1 stuff
  state=2; after(N, do_lcd); return; // set next state, use after to delay execution by N milliseconds, exit do_lcd()
case 2: // do state 2 stuff
  state=3; after(M, do_lcd); return; // set next state, use after to delay execution by M milliseconds, exit do_lcd()
} }

The busy waiting version of do_lcd() was:
do_lcd() {
  // do state 1 stuff
  delay(N);
  // do state 2 stuff
  delay(M);
} }

The do_lcd() function is ugly, but it is working without busy waiting. If all Arduino users are annoyed by event handling uglyness, they are ready for phase 3 of the RTOS brainwash program:

Migration step 3: Show that multi-threading does look better
----------------------------------------------------------------
Multi threading is dangerous. The worst thing you can do with busy waiting is to freeze everything up. The worst thing in multi threading is that you literally tear apart your machine because thread 1 says "go left" and thread 2 says "go right" as fast as time-division multiplexing can do. But, if you have multi-threading and a multi-threaded version of delay(), the do_lcd() function will no longer look ugly, and you are still out of busy waiting land:

do_lcd() {
  // do state 1 stuff
  delay(N); // the multi-thread version of delay() will switch to another thread or will wait in the operating system
  // do state 2 stuff
  delay(M); // the multi-thread version of delay() will switch to another thread or will wait in the operating system
} }

I bet, everybody will be happy about a keyboard() lib. And they will learn that "fu**ing" event crap, because using a lib is better then writing a lib. And everybody who uses keyboard() will use after() and every(). And now the future is open: If you can live with the event programming uglyness, you never have to go to multi-threading land. Maybe you remember the X windows system, the GUI for UNIX and Linux? Well this was all done by using event programming, and it was done a long time before multi-threading was invented.
Title: Re: Does Arduino need a real-time scheduler?
Post by: Se3ker on Dec 16, 2013, 01:19 am

My opinion: Arduino needs a better delay() function  :smiley-mr-green:

The question RTOS or no RTOS does just heat up the discussion, but does not help the migration. This is my approach:
I want a better delay() function - this is a feeling. If I introspect this feeling I recognize
- it is a waste of energy for me if delay() is just doing busy waiting.
- I want/I need to do several things in parallel, but delay() does not allow me to do it.
- I do not want to throw everything away because of a bigbang RTOS solution, I want to go step by step from here to there.

As a seasoned software engineer I have seen again and again that migration is the key. Here are my suggestions:

Migration step 1: Introduce event handling
--------------------------------------------
The Arduino "core library" has the loop() function. An event handling Arduino program will have one only statement in the loop() function, the do_event() function. The do_event() function has to look after the events and has to service the events.
A useful vehicle to show that event handling is useful should be a keyboard scan library that provides its services within an event handling framework. The typical pocket calculator keyboard has arranged the keys between row and column lines. A 12 button keyboard needs 3+4=7 scan lines. The keyboard scan function is not trivial. Everybody expects to get:
- key debouncing
- key repeat (repetition)
- 2-keys-rollover (Rapid typists may sometimes inadvertently press a key before releasing the previous one)
- no electrical shortcircuit if two keys are pressed simultaneously
To implement these functions, a timer is needed. The keyboard lib should provide a timeout functionality to the user. The function set of the keyboard lib is:
button_register(): combine an event (key press) to a function (eventhandler)
after(): after N milliseconds call a function
every(): every N milliseconds call a function
do_event(): check if there is a pending event and execute the eventhandler function. Needs to be called again and again within loop().

A simple Arduino program that fits nicely the event paradigma is a math trainer program. The Arduino displays a math question like "1 + 2 = ?" and the user should enter the correct answer within a time limit. In an event program there will be a keyboard event for every digit and a timeout event. If the user presses the digit "3", the button-3-event function will say "correct answer". The other button-event functions will say "wrong answer". And the timeout event-function will say "timeout".

Migration step 2: Create a event-driven LiquidCrystal lib
----------------------------------------------------------
Everybody is using LiquidCrystal. But LiquidCrystal uses delay() and busy waiting is the nemesis to all real time programming. Thanks to the keyboard lib above, we have one-shot timer. And an event driven LCD output function does look like this:
do_lcd() {
switch (state) {
case 1: // do state 1 stuff
  state=2; after(N, do_lcd); return; // set next state, use after to delay execution by N milliseconds, exit do_lcd()
case 2: // do state 2 stuff
  state=3; after(M, do_lcd); return; // set next state, use after to delay execution by M milliseconds, exit do_lcd()
} }

The busy waiting version of do_lcd() was:
do_lcd() {
  // do state 1 stuff
  delay(N);
  // do state 2 stuff
  delay(M);
} }

The do_lcd() function is ugly, but it is working without busy waiting. If all Arduino users are annoyed by event handling uglyness, they are ready for phase 3 of the RTOS brainwash program:

Migration step 3: Show that multi-threading does look better
----------------------------------------------------------------
Multi threading is dangerous. The worst thing you can do with busy waiting is to freeze everything up. The worst thing in multi threading is that you literally tear apart your machine because thread 1 says "go left" and thread 2 says "go right" as fast as time-division multiplexing can do. But, if you have multi-threading and a multi-threaded version of delay(), the do_lcd() function will no longer look ugly, and you are still out of busy waiting land:

do_lcd() {
  // do state 1 stuff
  delay(N); // the multi-thread version of delay() will switch to another thread or will wait in the operating system
  // do state 2 stuff
  delay(M); // the multi-thread version of delay() will switch to another thread or will wait in the operating system
} }

I bet, everybody will be happy about a keyboard() lib. And they will learn that "fu**ing" event crap, because using a lib is better then writing a lib. And everybody who uses keyboard() will use after() and every(). And now the future is open: If you can live with the event programming uglyness, you never have to go to multi-threading land. Maybe you remember the X windows system, the GUI for UNIX and Linux? Well this was all done by using event programming, and it was done a long time before multi-threading was invented.



Well, I don't know exactly what you mean by this, do you condone one way or the other, but I for sure am not a software architect and there for will not write my own state machine, I expect this to be handled by the framework so I can focus on my logic level. I expect to write a couple of lines of code to get a couple of lines of logic nothing fancy nothing hard.
Title: Re: Does Arduino need a real-time scheduler?
Post by: westfw on Dec 16, 2013, 01:51 am
Yeah, I think we need to stop thinking of the implementation, and start thinking about what the API should look like if it's going to be usable by "The Arduino Community."

timer mytimer;
void timerSet(mytimer, timeInms)
void timerSet(mytimer, hours, int minutes, int seconds)
boolean timerExpired(mytimer)
long timerTimeLeft(mytimer)

alarmClock myclock
alarmClockSet()
alarmSet(endTime)

In this case "timers" implement a delay for some duration, while alarmclocks implement checking whether some absolute time has ocurred.
Title: Re: Does Arduino need a real-time scheduler?
Post by: jimford on Dec 18, 2013, 02:07 pm
If you want to cater for a hobbyist like myself with a non-software background, then it's essential to provide good clear documentation.

I for one will alway choose a library of any complexity that has good clear documentation, even at the expense of overlooking a possible more suitable one where I'm expected to wade through the code to deduce its workings.

My interest in scheduling is because I'm reading a few sensors with an Arduino.. I'm using Dr Liu's 'phi_prompt' menu library to select and display the values on an LCD. Unfortunately the phi_prompt  library sits in loops waiting for user input, during which the sensors aren't read, so I'm looking for a way round the problem and suspect the way forward may be some sort of scheduling system.  I'll look at NilRTOS (thanks 'fatlib16'!), but I've not seen a big fat PDF file describing it, so I may not get far!

Jim
Title: Re: Does Arduino need a real-time scheduler?
Post by: mrsummitville on Dec 19, 2013, 02:04 am

Yeah, I think we need to stop thinking of the implementation, and start thinking about what the API should look like if it's going to be usable by "The Arduino Community."

timer mytimer;
void timerSet(mytimer, timeInms)
void timerSet(mytimer, hours, int minutes, int seconds)
boolean timerExpired(mytimer)
long timerTimeLeft(mytimer)

alarmClock myclock
alarmClockSet()
alarmSet(endTime)

In this case "timers" implement a delay for some duration, while alarmclocks implement checking whether some absolute time has ocurred.


How about simply  "sleep(x);"  like ...

code here;            /* Last line to run before sleeping     */
sleep(100);          /* Thread goes to sleep here             */ 
code here;           /* Thread wakes up here after delay */

sleep(); causes THIS thread to release the CPU back to the RTOS.
Other threads can run while this thread is sleeping.
When the Timer Expires then the Interrupt allows the RTOS to "wake up" the sleeping thread.

Title: Re: Does Arduino need a real-time scheduler?
Post by: westfw on Dec 19, 2013, 09:57 am
Quote
sleep(); causes THIS thread to release the CPU back to the RTOS.

No, no!  If we had an RTOS with threads we can just use delay() (naming conflicts aside.)
I want to know what APIs we can use FROM NOTHING, because I think basic RTOS concepts are too difficult.
Title: Re: Does Arduino need a real-time scheduler?
Post by: mrsummitville on Dec 19, 2013, 02:24 pm

Quote
sleep(); causes THIS thread to release the CPU back to the RTOS.

No, no!  If we had an RTOS with threads we can just use delay() (naming conflicts aside.)
I want to know what APIs we can use FROM NOTHING, because I think basic RTOS concepts are too difficult.


OK, Use the word "delay(x)" .
It does not matter what you name.
Although, many RTOS use sleep() as it more accurate description of what actually occurring.
It was the concept that: "THREADS SHOULD NOT BLOCK".
This is one difference between multi-threaded vs. single thread.
And is the best way to utilize CPU resources.

Also, if you want to use a Single Thread in RTOS then you can still do that.
It is not required that you make you App multi-threaded.
Multi-Threading Is an option that is available if you have the skill set to use it.

Much like using the more advanced "Interrupt Driven" Threads we can use now.

Title: Re: Does Arduino need a real-time scheduler?
Post by: z900collector on Jan 01, 2014, 03:09 am
Happy New Year everyone!

I would like to weigh into the debate on this one... with a resounding YES we need an RTOS, however I recognize that the majority of users will not know how to implement it due to a procedural mindset and thats not a bad thing but for the rest of us, the ability to have tasks running and doing stuff is perfectly natural.

To date the CHibios port looks great, FreeRTOS might have too much over head, but one thing we will need in it from day 0 is a fully working Ethernet Library and all the existing libraries will need to be task/thread safe.

I'm old school, started coding back in 1976 but I also can readily adapt to change when change is good, I do a lot of work in message queuing systems at present and write a lot of C++ code. To me kicking off jobs across 100's of queues so tasks are decoupled and running in various states of parallelism is perfectly logical, but for the majority of programmers they just dont get it... so the RTOS would need to be so well integrated you dont have to think about it. Then when the skill level of the coder rises he/she will start to use tasks and think in terms of lots of stuff happening at once.

I have had a lot of exposure to hardware and software for just about every industry and spent many years doing battle in the embedded space...
I think for the Arduino to jump into the commercial world it needs an RTOS. and it is a commercial product if you look at just how much stuff is available for it... my other bug bear is the mounting holes... if you design an board, please make the holes line up! (Rant over).

Sid