SCoop - multitask and Simple COOPerative scheduler AVR & ARM

fat16lib:
I expect Arduino to do something extremely simple. Arduino users are mostly hobbyists looking for instant gratification so learning how to properly use an RTOS won't be appealing to most users. Even a system like SCoop has too many features and is too complex.

That's the reason why I wrote leOS, a very simple scheduler that do little jobs in background, using an interrupt-driver mechanism.
No preemptive, no cooperative... a "task" (or job, a better therm), waits until its time is reached and then the scheduler calls it. The job then runs until it has done its work and then it leaves the control back to the ISR, that terminates its work and returns the control at the main loop. A supervisor system checks if a task freezes, in such case it resets the microcontroller (useful in case your circuit runs in a place not easily accessible).

No complicated setups for the user, no complicated methods to add/modify/pause/delete jobs.
No separated editors/IDEs to use, just a simple library to include and use.

PS:
I would say a couple of words.
I followed this thread and the other ones where you, fat16lib, partecipated talking about RTOSs and I appreciated your ideas and your knowledge.
Thanks for every work you spent trying to explain us the oceans about RTOSs :sweat_smile:

leo72,

I am sure efforts like leOS will work for some problems. Maybe you have even found a Silver Bullet in the Holy Grail.

No preemptive, no cooperative... a "task" (or job, a better therm), waits until its time is reached and then the scheduler calls it. The job then runs until it has done its work and then it leaves the control back to the ISR, that terminates its work and returns the control at the main loop. A supervisor system checks if a task freezes, in such case it resets the microcontroller (useful in case your circuit runs in a place not easily accessible).

This approach is not general enough for many embedded multitasking situations. RTOSes aren't complex unnecessarily. Embedded systems deal with devices in ways that users of a PC, Mac, or Linux never see.

Of course simple apps don't need multitasking. Arduino is doing fine for lots of things. Adding multitasking to Arduino in a way that is reliable, simple, and easy for users to understand and apply will be tricky.

Here are features that you might look for when evaluating an RTOS. If you don't understand why an RTOS would have these things, you are not ready to write an RTOS replacement.

Abstract streams and channels
Binary semaphores
Condition variables
Core memory manager
Counting semaphores
Event flags
Hardware Abstraction Layer (HAL)
Heap allocator
I/O queues
Interrupts nesting
ISRs abstraction
Kernel base threading services
Mailboxes (message queues)
Memory pools allocator
Mutexes (priority inheritance)
Preemption
Priority based scheduling
Round robin scheduling
Static kernel
Synchronous messages
Threads registry
Timeout on wait primitives
Virtual timers
Zero latency interrupts

The question for Arduino is how much of this is needed internally to support sharing of resources and devices for multitasking and how much is presented to users in new APIs.

Every time I think about the current Arduino environment I find problems. A coop scheduler doesn't avoid sharing problems. If you put a yield inside existing libraries, two libraries like Enet and SD may access a shared resource like an SPI bus. Two threads could be writing to a files and trash the SD cache.

If you add preemption or run code for String in an ISR you will trash the heap. String uses malloc/free and these are not thread safe.

The exercise of porting ChibiOS and FreeRTOS to Arduino has opened my eyes to how primitive the current state is. Arduino is like an old DOS system or the original MAC OS. OS X is far away.

fat16lib,
I undestands the problematics that relies in a RTOS and I don't want to raise me up to those coding levels :sweat_smile:

I think that an RTOS is more complex than I know but think that a "real" RTOS on a little MCU like the Atmega328 can be too complex for the normal usage too. Things like Scoop or avr-os reserve 256 byte of SRAM for each task... in a product that contains only 2048 byte of RAM. Too expensive in therms of resource consumption.

I am very very interested in such projects (Scoop, avr-os, RTuinOS, FemtoOS) but IMHO I consider them too complex for newbie users (like me... :sweat_smile:).

Thanks for your reply :wink:

SCoop V1.2 is out and brings lot of goodies :slight_smile:

SCoop V1 brougth the idea of easy cooperative scheduler by extensively using OOP and by providing a (hopefully) comprehensive user guide.
Several objects class like SCoopTask SCoopTimer SCoopEvent and SCoopFifo brings good support for serious but simple multitasking. With the 3 other complementary libraries (IOFilter, TimerUp&Down) this brings a set of features that makes Arduino experience even simpler

V1.2 is very much faster and brings performance close to what we get with very basic scheduler like the "Android" one provided with arduino 1.51 for Arduino DUE, but with much more features!

the yield() routine has been optimized and is now providing a mechanism to switch to next task without coming back to the main scheduler routine which saves lot of CPU time. the result is 15us total switching time on AVR and an amazing 3.2uS on Teensy 3.0 !
lets be fair, the Android version (included in SCoop pack) is still 40% faster but it does nothing else than swiching :slight_smile:

The concept of delaying yield() by several micros seconds called "quantum time" is still in place and makes a huge difference with traditional cooperative scheduler as this brings the benefit of some CPU resource allocations to tasks, and some predictability in the cycle length, like we have with preemptive RTOS. The micros() routine have been optimized for Arduino UNO (by reusing some tips from the Teensy core) and this brings a very good 2.5us lead time to call yield() (AVR) and to check if it is time to switch or not to the nex task, so only <20% of a standard yield() which enable calling yield() everytime everywhere then.

New methods have been introduced to pause/resume timer, events or task, and to monitor their "state"

A new SCoopDelay object is introduced to rationalize timeout handling inside the library and it can be used in the main sketch (sort of simpler TimerDown)

last but not least, the SCoop pack is now provided with a copy of the standard Android Scheduler, ported for the AVR, and then compatible with both ARM or AVR platform and arduino < 1.5 (tested on 1.0.3) !
Then you have the choice : test and adopt the SCoop suite or quickly include the android scheduler.h and play multitask imediately also on AVR.

for mor details on the changes, please review the change log file as few but some important changes are NOT backward compatible with V1.1.1 including the basic defineTimer macro ...

every thing on GooGle code here:
https://code.google.com/p/arduino-scoop-cooperative-scheduler-arm-avr/

thanks for supporting us and providing your input or feedbacks

edit : change log in attachement
edit: all source code now on github : GitHub - fabriceo/SCoop at v1.2

scoop change log.txt (8.26 KB)

I am sure this is a stupid question..

I have SCoop operating well, EtherMega, no issues, clean and simple install.

However, whenever I introduce a call to a subroutine from any myTask::loop (say to call a common logging routine) the code reboots. This is not the case when subs are called from myTask::setup or from Timers.

Am I doing something wrong ;-p

rich..

Hello omegaman477
I d bet on a problem with a stack colision.

could you please try extending the size of the stack allocated to the task, say 512 and then trying again.
once it runs ok, you can reduce the size of the stack by monitoring it with the stackLeft() method.

if this doesnt work, then this might be link to the "3BYTESPC" but I would need a copy of .elf file to troubleshoot it

let me know and dont hesitate to attach or post your code, or PM if preffered

fabriceo:
Hello omegaman477
I d bet on a problem with a stack colision.

could you please try extending the size of the stack allocated to the task, say 512 and then trying again.
once it runs ok, you can reduce the size of the stack by monitoring it with the stackLeft() method.

if this doesnt work, then this might be link to the "3BYTESPC" but I would need a copy of .elf file to troubleshoot it

let me know and dont hesitate to attach or post your code, or PM if preffered

Increasing stack size had no effect. Where are the ELF files located, I will IM them and the Source to you.

Thanks.

Version 1.2, manual corrections and volatile

I see that a new version 1.2 has been posted on https://code.google.com/p/arduino-scoop-cooperative-scheduler-arm-avr/, thanks for that. I am just starting a new project and I think Scoop is the right tool for me and am going to try it.

I would like to propose some corrections to the manual, but I do not seem to find the source of it, which would make the job much easier: can I access it anywhere? In any case, where should I send proposal for changes?

By the way, one of the things I notice in the manual is the use of volatile in the very first example, the one on the first page. As far as I can tell, volatile is useless here, because the compiler will never optimise out accesses to this variable. That would happen if we had an infinite while loop instead of the loop entry point and if the scheduler were preemptive, but none of these two conditions are true here.

Hello!

I m sure you will get success in using SCOOP, plese let us know!

for the document, just give me your email adress by private message and I ll send you the original .doc file.

for the volatile count ... you are probably right in this example. But in general, when the variabe is used accross tasks, my experience is that you avoid loosing time in trouble shooting by putting them volatile , even in coopreative mode :stuck_out_tongue:

hope you succeed in your project

I sent you my mail address with a private message.

As far as volatile is concerned, you may be right that there are cases where this is necessary even with cooperative scheduling (even if I do not see on off the top of my head), but stating in the first example that this is necessary is confusing and adds unnecessary complexity and black magic.

Thanks for the excellent simple kernel which does not have many issues that come with preemptive ones when using standard Arduino libraries.

Now I don't have to use superloop to call state machines to handle different events. State machines are often hard to write and difficult to understand. Each task can now almost can behave as if it is an independent program, similar to rtos.
Hope to introduce to my project students.

As for the volatile, I suppose it is often better to be safe.

Thank you.

Thanks for your nice and very useful scheduler libraries.

After having worked with SCoop, I discovered that I was using no one of the many goodies provided, apart from stack printing. So I tried the SchedulerARMAVR and I discovered that it saves over 1500 bytes of program memory and 58 bytes of data memory with respect to SCoop, both of which are important to my project. Having to use dynamic allocation for the stack is a non-issue for me, as I allocate it during setup() and never free it.

The only thing I was missing is the printing of the size of the stack left free for the tasks. So I got down and wrote my own functions, which appear to work for me.

It is possible to make them slightly more compact by incorporating them into SchedulerARMAVR.cpp, I can provide a patch if you wish. I think they would be a useful addition to SchedulerARMAVR. Here they are:

// Use this rather than Scheduler.StarLoop() and save the pointer returned
// so that you can  pass it to stackUnused().
// This function relies on malloc to just give back the same pointer after free(),
// and on Scheduler.StarLoop() to do the first malloc() call to allocate the stack.
// If the SChedulerARMVR code is changed so that the above is not true, memory corruption will result.
byte *startLoop (SchedulerTask task, unsigned stackSize) {
  byte *stack = (byte *)malloc(stackSize); // the same allocation Scheduler.StarLoop will do
  // If we are being called from the main loop, let's fill our stack, too
  extern byte *__brkval;		   // set after malloc, it is the top of the heap
  if ((byte *)&stack > __brkval)	   // this function called from the main loop
    for (byte *end = (byte *)&end - 10;	   // leave space for a small interrupt routine
	 __brkval < end;		   // until end pointer smashes into the heap
	 *end-- = 0x55) ;		   // fill stack with fixed value
  memset(stack, 0x55, stackSize);	   // fill the stack's stack with fixed values
  free(stack);				   // now we free the space allocate with malloc
  Scheduler.startLoop(task, stackSize);	   // start the task
  memset(stack, 0x55, 4);		   // rewrite over memory dirtied by free()
  return stack;				   // stack base, to be passed to stackUnused()
}

// When not using the scheduler, call it from setup()
// Else, this work is done when calling startLoop(), so this function should not be used
void fillStack () {
  extern byte __heap_start, *__brkval;
  byte *beg = (byte*)(__brkval ? __brkval : &__heap_start);
  byte *end = (byte*)&end - 10;		// leave space for a small interrupt routine
  while (beg < end) *beg++ = 0x55;	// can't use memset here: it would use the stack
}

// Count bytes marked 0x55 from the stack base to the the local variables
// Should be called with either no arguments, for counting free stack on the main loop,
// or with two arguments, for the tasks.
unsigned stackUnused (byte *stackBase, unsigned stackSize) {
  byte *end = stackSize ? stackBase + stackSize : (byte *)&end - 10;
  byte *p = stackBase;
  while (*p == 0x55 && p < end)
    p += 1;
  return p - stackBase;
}

In order to use the code, you should use startLoop(task, stackSize) rather Scheduler.startLoop(task, stackSize), and save the return value, that is, the base of the stack of the task just created. Also, save the value of the stack size in a variable.

Then, you can call stackUnused() either from the main loop without arguments, or from wherever you want with two arguments: a task's stack base pointer and the task stack size. It returns the number of bytes on the stack that have been left unused for the main loop or for that task, respectively.

I've been looking for a simple task manager/scheduler for my project, and came across SCoop.

I did as I usually do first off with 'strange' libraries and tried to run the examples to get a 'feel' for SCoop.

I found that none of the examples would compile with the Arduino 1.5.4 IDE - they all gave multiple errors that appeared to be library related, but I didn't look any deeper.

Is there an 'issue' with SCoop and the Arduino IDE?

Jim

OK no response so far, so some details:

EXAMPLE 1
VERSION 1.2 NEW YEAR PACK 10/1/2013

Gives the following compilation errors:

Arduino: 1.5.4 (Linux), Board: "Arduino Duemilanove or Diecimila, ATmega328"

example1:7: error: expected constructor, destructor, or type conversion before ‘(’ token
example1:10: error: expected constructor, destructor, or type conversion before ‘(’ token
example1:13: error: ‘myTask2’ has not been declared
example1.ino: In function ‘void loop()’:
example1:14: error: ‘sleepSync’ was not declared in this scope
example1.ino: At global scope:
example1:17: error: expected constructor, destructor, or type conversion before ‘(’ token

This report would have more information with
"Show verbose output during compilation"
enabled in File > Preferences.

All the the other examples give similar errors, plus a whole load more!

Ideas, anyone - please?

(Come on 'fabriceo' SCoop's - your pet. You should be able to provide some input here! ;^) )

Jim

I have not tried, but it looks like <SCoop.h> is in fact not included.
As the message says, try to enable verbose output to get more information.
Make sure you have SCoop installed int he right place and enabled in the IDE.

fpoto:
I have not tried, but it looks like <SCoop.h> is in fact not included.
As the message says, try to enable verbose output to get more information.

The above message is verbose.

Make sure you have SCoop installed int he right place and enabled in the IDE.

It all appears to be installed in the correct place and appears in the IDE OK.

The only example I've managed to get to comple is the SCoopME template.ino. In fact I'm starting to use SCoopMe in my project and it shows some promise. I've abandoned SCoop! The 'basic sketch' on page 1 of the 'Scoop User Guide V1.2' doesn't compile for me either.

Do you know what the difference is between SCoop and SCoopME ? 'fabriceo' offers no clues and appears to have gone silent on this thread!

Thanks for the reply.

Jim

I had a look at the SCoopME sources, but without diving into it: it looks like a cleaned up version of SCoop, better written and probably smaller. I suggest you use that instead of SCoop, at least until fabriceo comes up and explains what are the differences.

In fact, I had tried SCoop and then witched to SchedulerARMAVR, which is much simpler, and which I am using right now, because I do not need all the goodies provided by SCoop, it is much smaller and has the same interface as the standard Arduino Scheduler library. The only thing is that I needed to measure the stack occupancy of tasks and I had to write those functions by myself (I posted them above).

fpoto:
In fact, I had tried SCoop and then witched to SchedulerARMAVR, which is much simpler, and which I am using right now, because I do not need all the goodies provided by SCoop, it is much smaller and has the same interface as the standard Arduino Scheduler library.

Interesting - I'll have a look at the SchedulerARMAVR library. Like you, my project uses very little of the facilities that SCoop/SCoopME offer and SchedulerARMAVR may suit.

There's precious little documentation for SchedulerARMAVR and there appears to only the Scheduler.startloop and yield() functions. Do you happen to know of anything else that will enable me to get a 'handle' on it, please?

I notice in the 'Multiple Blinks' example:

 // IMPORTANT:
  // When multiple tasks are running 'delay' passes control to
  // other tasks while waiting and guarantees they get executed.
  Scheduler.delay(1000);

I'm a bit confused as I thought it was yield() that passes control to the other tasks!

Jim

First look at the Scheduler library page, then at the page where Fabrice Oudert announces SchedulerARMAVR, you'll find many indications.

Anyway, if you use the 1.5.x library, the delay() function calls yield(). If you use the 1.0.x library, the delay() function does not call yield(), so you should use Scheduler.delay() rather than delay() to obtain the same effect.

Hello;

I m so sorry that I didnt pay attention to this thread in the past weeks... I was spending time on another fancy project more around electronic and DSP, but ok :slight_smile:

for some of you having tried Scoop V1.2, for one year ago I havent tried it myself on the early release of Arduino and I ll try to do so in the comming days. I ve just received also a pair of teensy 3.1 and a pair of sparkcore and will check that the lib is working with arduino 1.05 and teense 1.18 and may be on spark later

it is fair to use the SchedulerARMAVR instead of SCoop as it is smaller. I ve just put the Assembly code in it to make it compatible with AVR, credit goes to others and the android team;

still Scoop can offer a bit more features, especially the ring token handling and some time allocation to task, which at some point in your devlopement might be needed to optimize the CPU time given to a task

SCoopeME is a concept which is a rework of SCoop 1.2 but using object templates instead of macros.
also it is ported for TexasIntrusment MSP430 (at least I got it to work)
I ve missed time to finish it as I wanted, and to write proper documentation, but it s not bad and can be considered if you are an early adopter :slight_smile:

I still strongly beleive in cooperative scheduler concept and whish all the best to the 230+ people having downloaded it and the other to come

cheers and Happy new 2014