Request for feedback on messaging library design

Hi All,

I'm looking for feedback on a proposed design for a new messaging library to make it even easier to write PC software which communicates with a system containing an Arduino.

Here's an overview of the design: https://docs.google.com/document/d/1JCSjj6AWIs_lcogMreZ_ZsmJQlOdutBNCpggyXX6Sr0/edit?hl=en_US

I'm interested to know if there is already a library that does this (I found a few specifically designed for directly controlling the Arduino's pins, but none for generic message passing), and if other people would find this useful if I built it.

Regards, James McGill

Had a quick look, and it sounds interesting as this question comes up many times.

things and questions that came into my mind:

  • RPC recalls - http://tools.ietf.org/html/rfc5531 -

  • the statement : import shine.protocol.*; will not work for Arduino; #include <shine.h> might :wink:

  • output.set_sum(sum); how you generate set_sum member?
    do you intend to make a tool that compiles the protocol definition(s) into an arduino C++ class

  • support for ethershield in the future?

  • support for NewSoftSerial?

  • Can the Arduino take initiative? so python is listening?
    in other words is it symmetrical?

  • How to handle precision , think float? IEEE754 32 bit?

  • if I define 20 different messages are they all mapped upon INput & Output?

  • can you describe how to pass a complex struct e.g { string, int16, boolean }

  • what are the supported datatypes?

OK enough questions for now.

I concur, everything robtillaart wrote, me too. I would add that the Processing (i.e. Java) would be a top language to consider for the initial releases, as it is what the "artists" are using when doing interactive art. Python is a good top language, too, I see it in the questions of some people.

Hi robtillaart,

Thanks for the concrete feedback - replies are inline:

Thanks - I worry that this might be a bit verbose, but I’ll try and mirror the status codes at least.

  • the statement : import shine.protocol.*; will not work for Arduino; #include <shine.h> might

Good call. Specifically it would be more like #include “protocol.shn.h”. The header file is dynamically generated from the protocol.shn file at compile time. This is easy in AVR land (a new dep in the Makefile). I’m not 100% sure how to hook this into Arduino - perhaps someone can help me here.

  • output.set_sum(sum); how you generate set_sum member? do you intend to make a tool that compiles the protocol definition(s) into an arduino C++ class

Yes. Protocol files will be compiled to header files that can be included by Arduino (or AVR) projects. This means that the code can be efficient as many things can be resolved during code generation, rather than at runtime (e.g. casting).

  • support for ethershield in the future?

Absolutely. I think the only thing that would change is that larger packet sizes might make sense with Ethershield. I plan for packet sizes to be tuneable (depending on the memory requirements) so that’s easy.

  • support for NewSoftSerial?

Sure. In this case it might make sense for the transport channel to be independent of the rest of the protocol e.g.:

transport = AnythingThatCanTransferBits(9600, etc);
channel = Shine.channel(transport);

The transport would be a thin wrapper around any library that can send bits between two systems (Serial, Soft Serial, Ethernet, Infrared).

  • Can the Arduino take initiative? so python is listening? in other words is it symmetrical?

Yes - The Arduino can call methods on the PC too. The important thing is to identify which of the two endpoints has less memory so that it can run the memory-efficient version of the protocol. In the document I use the * operator before an RPC to indicate the direction of the call.

  • How to handle precision , think float? IEEE754 32 bit?

The data types I intend to support are int8, int16, int32, float, double (downcast to float if not supported), string, bool and repeated versions of these. On a PC → PC system I’d stick to always using int32 but for embedded development I think int8 and int16 are important. Floats would be IEEE floats - I think they are sufficiently ubiquitous.

  • if I define 20 different messages are they all mapped upon INput & Output?

No - Input and Output just happened to be the names of the messages I defined (perhaps poor choices). If you had message FooBar { } then your function would take an instance of FooBar. The first parameter is always the input, the 2nd parameter (if present) is an output to be filled in. You can define an unlimited number of messages and RPCs (until you run out of program memory on the Arduino).

  • can you describe how to pass a complex struct e.g { string, int16, boolean }

This would be defined as:

message Foo {
string name;
int16 age;
boolean isAwesome;
}

On the PC side you might generate this as:

foo = Foo(“James”, 99, True)

or

foo = Foo()
foo.set_name(“James”)
foo.set_age(99)

Over the wire it would be represented something like (in bytes):

1, ‘J’, ‘a’, ‘m’, ‘e’, ‘s’, ‘\0’, 2, 99, 3, 1

On the Arduino side you would access it as:

string name = foo.name();
etc.

  • what are the supported datatypes?

As above.

Thanks again - feel free to ask more questions! This has already helped me flesh out some of the ideas.

Hi Msquare,

I don't have any experience in Processing, but agree that it would be a good choice. I'm happy to document the protocol in more detail, and work with anyone who would be interested in implementing the protocol in another language while I work on the Python version.

Regards, James

Thanks, sounds like you really have a picture of it.

Good call. Specifically it would be more like #include "protocol.shn.h". The header file is dynamically generated from the protocol.shn file at compile time. This is easy in AVR land (a new dep in the Makefile). I'm not 100% sure how to hook this into Arduino - perhaps someone can help me here.

The IDE must be restarted if new libraries are installed, so I propose to use a separate makefile to compile the protocol.


Sending structs: There must also be meta info about the struct e.g. what is the max size of string or an array. Especially in embedded this is important for allocation or RAM might become a "nightmare" :)

Rob

yet another thought - http://en.wikipedia.org/wiki/JSON - to define and document the messages…

If you pursue a python implementation, a pyro proxy would be awesome!