Cosa: An Object-Oriented Platform for Arduino programming

@g_ad

That is actually exactly the behavior it "should" have right now when the line is too long. The input buffer is limited to 64 characters (in the current version). It is a configuration variable. The error handling could be improved. Please see the configuration constants: https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/Shell.hh#L117

I will see what I can do about that in Shell::run() as the too long command lines will result in multiple gets() calls. These will have to be ignored and reported as an error.

Latest update; The script handling will allow parameter binding (standard shell script syntax, $1 for parameter/option one, etc). Please see the updated CosaShell.ino/Commands.cpp. The script blink will require a delay time parameter (in ms).

Cheers!

@kowalski

kowalski:
That is actually exactly the behavior it "should" have right now when the line is too long. The input buffer is limited to 64 characters (in the current version). It is a configuration variable. The error handling could be improved. Please see the configuration constants: https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/Shell.hh#L117

I will see what I can do about that in Shell::run() as the too long command lines will result in multiple gets() calls. These will have to be ignored and reported as an error.

Thank you Mikael, please treat my posts informational, by any mean not as a change request, I only shared it as result of the crash testing :wink: and if it can help you with the development that's good, otherwise just ignore it :). I'm happy to do some beta/stress testing :wink:

I think I'm going to capture the uart in the main loop and pass the string to shell when the command line is ready to be processed so I will not block the main loop while waiting for the command line, this way I can quite easily test the string before passing it to shell to avoid multiple gets() calls :).

Kind regards,
Adam

@kowalski

After the latest update the CosaShell example doesn't compile:

In file included from /Commands.h:27,
                 from Commands.cpp:21:
... \arduino-1.0.5-r2\hardware\Cosa\cores\cosa/Cosa/Shell.hh:121: error: ISO C++ forbids initialization of member 'ARGV_MAX'
... \arduino-1.0.5-r2\hardware\Cosa\cores\cosa/Cosa/Shell.hh:121: error: making 'ARGV_MAX' static

Kind regards,
Adam

@g_ad

Update available that fixes that for Arduino 1.0.X.

BW: It is better to use the github issues list for this type of problems.

Cheers!

A new screenshot of the Cosa Shell example sketch. There are a number of new commands; dump memory, play tone, repeat command lines, etc.

The options parser makes it really easy to write action functions with multiple options and variable number of arguments. Below is a snippet from CosaShell Commands. It implemented the memory dump command which has radix options, memory address and block size:

static const char DUMP_NAME[] __PROGMEM = 
  "dump";
static const char DUMP_HELP[] __PROGMEM = 
  "[-b|-d] POS [SIZE] -- dump memory";
static int dump_action(int argc, char* argv[])
{
  IOStream::Base base = IOStream::hex;
  uint32_t addr = 0L;
  size_t size = 256;
  char* option;
  char* value;
  char* sp;
  int ix;
  while ((ix = shell.get(option, value)) == 0) {
    if (strcmp_P(option, PSTR("b")) == 0)
      base = IOStream::bin;
    else if (strcmp_P(option, PSTR("d")) == 0)
      base = IOStream::dec;
    else 
      return (-1);
  }
  if (ix < argc) {
    addr = strtoul(argv[ix++], &sp, 16);
    if (*sp != 0) return (-1);
  }
  if (ix < argc) {
    size = strtoul(argv[ix++], &sp, 10);
    if (*sp != 0) return (-1);
  }
  if (ix != argc) return (-1);
  cout.print(addr, (void*) addr, size, base);
  return (0);
}

The first section uses Shell::get(option, value) to parse the given options. The next section handles possible parameter; memory address to dump from (default 0), and size of memory block (default 256). For more details please see https://github.com/mikaelpatel/Cosa/blob/master/examples/Sandbox/CosaShell/CosaShell.ino.

The Cosa Flash File System (CFFS) shell has also been updated with some new commands:

For more details please see https://github.com/mikaelpatel/Cosa/blob/master/examples/Sandbox/CosaCFFSshell/Commands.cpp

Cheers!

@kowalski

kowalski:
Update available that fixes that for Arduino 1.0.X.

BW: It is better to use the github issues list for this type of problems.

Thank you very much and I'm sorry for not reporting it on githube, you've mentioned it before :blush:

Now I want to mention something about CosaShell functionality and please correct me if I should do this on github as well, this is not so much about error/bug-issue. Btw - what should be reported on github and what on forum?

I want to mention something about CosaShell functionality.
In my professional life I deal with a lot of different devices through the console-rs232 and I'm using Putty on my Windows7; because of that Putty is my preferred way for serial monitor on Arduino instead of the built-in one.
Yesterday I was going to test the new Shell library and to my surprise I got nothing on my Putty apart from the prompt, everything was working with the built-in serial monitor so the conclusion was very simple - the Putty doesn't use LF as line termination, it uses CR. I tried to simulate other terminals and all of them interpret 'enter' as CR.
The above means that using the library in real development might be tricky and require code changing for at least Windows+Putty users :confused:
I'm far from requesting any change but if you would like to do some correction then instead of checking for LF line termination it could be checked for LF or CR.
Another problem is with sending line termination from the board to external terminal - all the other devices I used to work with definitely use CR, maybe they send CR+LF but to proof that I would have to capture the whole stream, not just the visible characters, however I can see that when I run the examples from the library that is using LF, then that is exactly what I can see on the terminal - the new line doesn't start in column 0 but stays in the same column as the line above.

As usual, if the above doesn't make sense please ignore it :slight_smile:
Kind regards,
Adam

@g_ad

Hi Adam. I use this topic of updates on the development of Cosa. There are questions from users and as other forums users mostly help each other. These questions also get mixed up in this topic.

Bugs, errors and feature requests should be posted on the Issues list on github. In general I do not add new boards, modules, devices, etc, that I cannot test or have my own project for. Pull requests that cannot be tested are not merged (of the same reason).

Now the more interesting question; handling of NEWLINE. Linux LF, Mac CR and PC both CR+LF, and others. Newline - Wikipedia

I did some tests with configuration of Putty and the Terminal window setting "Implicit CR in every LF" (and local echo auto) fixes one side of the issue. The other will require adding support for translation (as in many Linux tools). The simple fix was to translate CR to LF in gets().

The more elegant solution is to add a translation setting to IOStream::Device and default to a common standard. The NEWLINE handling needs to be abstracted to all type of devices (e.g. terminal, network, files, etc).

I do not use Putty and I work mostly on Linux/Ubuntu. To get the fastest possible development turn-around I have actually added a build script (modified Arduino-Makefile) and miniterm for serial monitor (also modified). This allows build of only what has changed, upload and starting the serial monitor as a single command. The Arduino IDE build is 3-5 times slower in build. A lot of unnecessary operations are performed as it does not use make.

Cheers!

@kowalski

kowalski:
@g_ad

Hi Adam. I use this topic of updates on the development of Cosa. There are questions from users and as other forums users mostly help each other. These questions also get mixed up in this topic.

Bugs, errors and feature requests should be posted on the Issues list on github. In general I do not add new boards, modules, devices, etc, that I cannot test or have my own project for. Pull requests that cannot be tested are not merged (of the same reason).

Ok, I think I got it, I'll learn how to use github.

kowalski:
Now the more interesting question; handling of NEWLINE. Linux LF, Mac CR and PC both CR+LF, and others. Newline - Wikipedia

I did some tests with configuration of Putty and the Terminal window setting "Implicit CR in every LF" (and local echo auto) fixes one side of the issue. The other will require adding support for translation (as in many Linux tools). The simple fix was to translate CR to LF in gets().

The more elegant solution is to add a translation setting to IOStream::Device and default to a common standard. The NEWLINE handling needs to be abstracted to all type of devices (e.g. terminal, network, files, etc).

Yes, I know the "Implicit CR in every LF" and I have it enabled for the arduino serial monitor, the problem appeared only when I've tried to use the Shell library with Putty, so yes the real problem is one direction, however the "Implicit CR in every LF" would be probably not elegant solution for the final device. I can change the part of the code for myself I suppose but wanted to share with you the finding in case you would like to implement it in the library.

The more I think of what I want to use, the more convinced I am that I will do some capturing of the Uart input, maybe implementing backspace and echo + eventually the CR to LF conversion on the main loop and then passing the cmd string to the shell. At the moment I'm still testing and learning from your code.

Thank you,
Adam

The news on the latest updates in Cosa.

  1. IOStream
    The IOStream class has a new member function to allow simple command line reading; IOStream::readline(buf, size). It will read available characters and append to the given string and return the buffer pointer when the line has been completed. The function will handle backspace, carriage-return and line-feed.

  2. Shell
    The command line support is now completed. The Shell::run() member function will handle the read, parse and execution of command lines. The function is non-blocking and should be called in the loop(). Below is a screenshot of the example sketch CosaShell. This sketch shows how to setup the shell for serial input, run the top-loop, and how to capture idle time.

    As the Shell class uses the new IOStream::readline() function all echo and backspace is handled by this function. Below is a screenshot of using PuTTY with the shell. This is the default PuTTY setting (on Ubuntu).

  3. Refactoring
    a. SLEEP
    The macro SLEEP (Cosa/Types.h) has been removed and the function sleep() should be used instead.
    b. IOBuffer::gets()
    This member function has been removed and the default implementation (IOStream::Device) is used instead.
    c. IOStream::print(const void *ptr, size_t size, Base base, uint8_t max)
    This member function is used to dump memory block. An additional base variant is now available that will prefix each line with a source address (i.e. not the given buffers address).

  4. Versioning
    The Cosa project does not have explicit versioning as it is non-commercial and sharing but to make it simpler for users there are specific snapshots of source and documentation. For more details see the install description and the section "Download Versions". Cosa/02-install.md at master · mikaelpatel/Cosa · GitHub. Please note that using the build script will require some additional configuration. Please see Cosa/Cosa.mk at master · mikaelpatel/Cosa · GitHub and previous post on this issue.

Cheers!

@kowalski

kowalski:
The news on the latest updates in Cosa.
...

This is brilliant :), thank you for all your hard work.

Kind regards,
Adam

g_ad:
This is brilliant :), thank you for all your hard work.

@g_ad

You are welcome!

I think that the Shell support is now more in line with what you where expecting from the start :). The development and refactoring has a lot to do with your input and tests - THANKS. I wanted to start-off with a simple blocking version and move slowly towards non-blocking as in the "final" version. With your input the update and evolution was somewhat faster.

I specifically added the "idle" command to the example sketch (CosaShell) to demonstrate the non-blocking IO and power down sleep that is going on in the background. There are more to be done on scripts; use EEMEM for storage, allow definition as commands, etc, but this will do fine for the release.

Thanks again for collaborating!

@kowalski

kowalski:
You are welcome!

I think that the Shell support is now more in line with what you where expecting from the start :). The development and refactoring has a lot to do with your input and tests - THANKS. I wanted to start-off with a simple blocking version and move slowly towards non-blocking as in the "final" version. With your input the update and evolution was somewhat faster.

I specifically added the "idle" command to the example sketch (CosaShell) to demonstrate the non-blocking IO and power down sleep that is going on in the background. There are more to be done on scripts; use EEMEM for storage, allow definition as commands, etc, but this will do fine for the release.

Thanks again for collaborating!

Thanks a lot, this is indeed something I was aiming for and I didn't expect anyone doing the hard work for me ;). I'm very happy that I was able to help somehow, hopefully more to come, maybe one day will be able to contribute some code more than ideas :).
I already looked at the code and figured out the "idle" :slight_smile: but still need to understand all around the "power down sleep" - I didn't yet think about the power management.

You are a star !!! :slight_smile:

kowalski:
The news on the latest updates in Cosa.

  1. IOStream
    The IOStream class has a new member function to allow simple command line reading; IOStream::readline(buf, size). It will read available characters and append to the given string and return the buffer pointer when the line has been completed. The function will handle backspace, carriage-return and line-feed.

Does this mean that I can use the RFM69/nRF24 also to handle 'shell' commands? In other words I can easily create a serial to LAN/WiFi/Radio translator/gateway??

BTW: You seem to be working om my secret COSA wish list I have while developing prototype hardware. It's magic ]:smiley:

MarsWarrior:
Does this mean that I can use the RFM69/nRF24 also to handle 'shell' commands? In other words I can easily create a serial to LAN/WiFi/Radio translator/gateway??

Not really there yet. You need an IOStream::Driver that will use a Wireless Driver for transportation. There is a mockup https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/IOStream/Driver/WIO.hh that does output only. See the example sketch Cosa/CosaWirelessIOStream.ino at master · mikaelpatel/Cosa · GitHub. With input handling (i.e. recv) and buffering in the Shell you have all the necessary components for remote commands (at least on the NRF24L01P which has built-in acknowledgement and retransmission).

I am actually working on a telnet and http service that uses the shell. Seems like an interesting challenge. See how much of that ends up in the Cosa source.

MarsWarrior:
BTW: You seem to be working om my secret COSA wish list I have while developing prototype hardware. It's magic ]:smiley:

I have suspected that we have somewhat the same agenda :slight_smile:

Cheers!

Some news on the latest updates in the Cosa project:

  1. New pin index to Cosa pin symbol mapping.
    In many example sketches there has been a need to map between a pin index to the program level pin symbol. There are two mapping tables available. See Cosa/Board.hh at master · mikaelpatel/Cosa · GitHub

  2. More accurate delay when using the RTC class (timer that implement micros() and millis()).
    The Cosa global delay() function is a function pointer and can be redefined. The default implementation uses busy-spin. When the Watchdog is used the Watchdog::delay() is installed instead. This gives low power mode during the delay. The Watchdog accuracy is at most 16 ms tick. When the RTC is used it will install the RTC::delay() function which also gives low power mode but also better accuracy (1 ms). The order of calling Watchdog and RTC begin() defines which version is used.

  3. Telnet Shell example sketch.
    The CosaShell example sketch has been integrated with the Telnet Server sketch to show how to communicate with the shell over Ethernet. Below is a screenshot with left) build, upload and serial monitoring, and right) telnet connection and commands to the shell.

    For more details see https://github.com/mikaelpatel/Cosa/blob/master/examples/Ethernet/CosaTelnetShell/CosaTelnetShell.ino#L108

Cheers!

Some performance measurements on the Cosa Telnet Shell and a short update:

  1. CosaTelnetShell performance
    The test bench is an Arduino Mega 2560, Ethernet Shield and a D-Link DWL-810+ Ethernet Wifi Bridge. Connecting through a Wifi Router and back to a laptop (HP Probook 4540s with Ubuntu 12.04 LTS).

a. Max burst string output
To measure the performance of pure text output from the sketch the command "help" can be used. It will print the command help text. Using the command "repeat 1000 help" will generate a longer period to measure. The measured throughput on the laptop is 60-64 K received bytes per second for this test case (using GNOME Terminal/telnet and PuTTY).

The max performance when filling buffers on the W5100 is 250 KBps (8 MHz SPI, 4 byte command to transfer 1 byte data, 1/4 MBps). This is excluding chip select and all the commands necessary to initiate and check status for the W5100 SEND command.

b. Max analog sample rate
The command "analogread a4" will sample the analog pin a4 and print the value to the io-stream (socket). Again the repeat command can be used to find the upper limit of sampling, converting and sending per second; "repeat -t 2500 analogread a4" will measure the time to execute 2500 analogread. The max rate is between 2000-3000 samples per second (sample values are converted to text, printed to the iostream/socket and sent over Ethernet/Wifi).

The variation is due to other traffic and drops on Wifi but also the sample value as the text conversion time depends on the value. Please note that the max sample rate at 112 us per sample is 8900 samples per seconds. And that the above results is for a pure sequential implementation without interleaving. It is possible to, for instance, run number conversion to text in parallel with the sampling (ADC). This could give as much as 4000-5000 samples (in textual format) per second over Ethernet/Wifi from the Arduino board.

c. New example shell commands
i. Command "pinmode" to get or set the pin mode; input, output and pullup. The command "pinmode all" will print the pin mode for all pins.

ii. The CosaShell/CosaTelenetShell are mainly to demonstrate how to use the Cosa command line support (class Shell) but may also be used as debugging tools; check hardware connections, etc. Two new commands have been added to allow scanning of 1-Wire and I2C (OWI and TWI).

iii. Setting date/time and accessing the Cosa RTC timer. The command "date" may be used to display or set the current date/time.

2. New SPI function; clock()
The new static SPI function clock() will map from frequency to the SPI clock rate setting. This allows device drivers to be written independent of the MCU system clock setting and knowledge of the SPI rate command. A SPI device driver only needs to supply the max frequency required on the bus (for the connected device).

  1. Boosting W5100 performance
    The W5100 device driver uses the new interleaving SPI API (spi.transfer_start/next/await) so that memory access (load/store) can be executed in parallel with SPI transfers. This gives a performance improvement of 10-20%.

  2. Bug fixes
    a. Analog mux setting on Mega. Failed to read bandgap correctly after using higher analog pin numbers.
    b. Improved W5100::Driver::flush() robustness when client or server disconnects.

Cheers!

Next step in integrating Cosa Shell with Ethernet protocols; A HTTP Server that runs the command line handler.

This HTTP server is simply put together by filtering the HTTP query and passing it to the same command handler as in the previous examples (telnet and serial). Below is a snippet from the request handler:

void 
WebServer::on_request(IOStream& page, char* method, char* path, char* query)
{
  static const char header[] __PROGMEM = ...
  static const char footer[] __PROGMEM = ...
  UNUSED(method);
  UNUSED(path);
  int res = -1;
  page << header;
  if (query != NULL) {
    char c;
    for (char* qp = query; (c = *qp) != 0; qp++)
      if (c == '&') *qp = ' ';
    res = shell.execute(query);
  }
  if (res != 0) page << PSTR("illegal query") << endl;
  page << footer;
}

https://github.com/mikaelpatel/Cosa/blob/master/examples/Ethernet/CosaShellWebServer/CosaShellWebServer.ino

The command line is the URL query where space between options and parameters are replaced with ampersand.

Cheers!

I see the Teensy 2 is supported. How about the Teensy 3.1?

Thanks,
Jim

AverageGuy:
I see the Teensy 2 is supported. How about the Teensy 3.1?

Hi Jim!

I am sorry to say that the Teensy 3.1 is currently not supported. It is actually a totally different architecture and has much more resources (memory, DMA, etc) which would allow a very different approach to building small to medium scale embedded systems with both RTOS and library support.

Cosa is very much tailored for the AVR and very limited memory resources. On the bright side there is only a hand full classes that are accessing hardware registers etc. The Cosa HAL :).

I do have a couple of the Teensy 3.1 boards but have not yet come around to start playing with them. The Teensy 3.1 Arduino core could be an excellent source for kick-starting the porting of Cosa to ARM/SAM/Teensy 3.1.

Cheers!

A short update on the latest development in the Cosa project.

  1. CosaShell
    The command line support has been extended with privilege level definition and checking. The built-in levels are GUEST, USER and ADMIN. The Shell::execute() member function will check that the privilege level setting before calling action functions.

    The action function may also perform further checking of privilege level depending on options and parameters or even program state. See Shell::is_privileged().
static int date_action(int argc, char* argv[])
{
  if (argc == 3) {
    if (!shell.is_privileged(Shell::ADMIN)) 
      return (Shell::PERMISSION_DENIED);
    uint32_t value;
    time_t now;
    char* sp;
    value = strtoul(argv[1], &sp, 10);
    if (*sp != '-' || value < 2000 || value > 2099) 
      return (Shell::ILLEGAL_COMMAND);
    ...
}

Please see the example sketch for more details on how this can be used: https://github.com/mikaelpatel/Cosa/blob/master/examples/Sandbox/CosaShell/Commands.cpp#L711 and https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/Shell.hh.

  1. Boosting build performance
    Cosa can be used with the Arduino IDE and the makefile based build support. The latest update introduces parallel build jobs to reduce build time further. Below is a comparison of the build time.

    The Cosa build dramatically reduces the time to compile the core library but also rebuilding sketches after editing. The measurements are performed on a HP ProBook 4540s (Intel® Core™ i5-3230M CPU @ 2.60GHz × 4) with 60 GB SSD , Ubuntu 14.04 LTS (32-bit). Please note that the version 1.5.7 is with link time optimization and extra compiler checking. The Cosa build improvement compared to the Arduino IDE build is approx. X3 for Cosa core library build and over X8 for sketch rebuild. Please note that the Arduino 1.5.7 core source code is approx. 8 KLOC (commented) while the Cosa core is over 62 KLOC (commented) and includes all library functions and support for ATtiny, Mighty, Mega, and many clone boards all in the core library.

  2. Telnet Server support class
    The Telnet class has be refactored to better support servers based on the Shell class. The Telnet::Server class will handle client connect/disconnect and call a set of virtual member functions on events such as data available. Please see https://github.com/mikaelpatel/Cosa/blob/master/examples/Ethernet/CosaTelnetShell/CosaTelnetShell.ino#L54 and the interface https://github.com/mikaelpatel/Cosa/blob/master/cores/cosa/Cosa/INET/Telnet.hh#L38.

Cheers!