Variable-length strings (or character arrays).

I wish to be able to send sequences of plain text commands to an arduino to access its functionality (by conditionally branching to any internal functions with any appropriate parameters). Commands could be things like:

SendIR (type) (size) (code) Relay (#) (On/Off) (delay) WriteSD (file) (entry)

or basically anything else that arduino may be capable of doing.

So I need to read incoming serial characters looking for a specific 'start' delimiter, then add subsequent characters to a 'command string' buffer until the next start delimter is reached, then parse the resulting variable-length 'command string' for valid keywords, then branch and action it as appropriate.

The overall command+parameter length is not predictable, but could be limited to a maximum length of 200 characters to give about 2 lines worth of text. Any additonal newly arriving characters need to stack in a queue for subsequent parsing after the current 'command string' has being actioned.

I suspect a competent programmer wouldn't have too much difficulty doing something like this, but unfortunately I'm not even at the level of incompetent programmer!

So I'm hoping someone may know of something existing which could be suitable, or else offer some words of wisdom regarding the best way for me to approach the problem (I've done a parser to pick out recognised commands and parameters, but I have no idea how to do the required buffering).

Thanks for any help.

I wish to be able to send sequences of plain text commands to an arduino to access its functionality

Try bitlash an arduino scripting language that does what you want already:- http://bitlash.net/

Thanks for responding, Mike.

Bitlash certainly looks interesting, and I've taken a look, but for my purposes it would be like using an elephant where a bicycle is needed... the jumbo would not be as useful as a bike, and would waste valuable resources and monopolise everything.

The irony is that is exactly the sort of arduino resources limitation that I hope to overcome with buffered variable-length strings - by providing a simple low-resource way for arduinos and ESP8266s to communicate and share their individual functionality with each other.

Imagine an arduino dedicated to multi-IO, and another dedicated to IR... then imagine the multi-IO node able to issue plain text serial commands for the IR node to recognise and respond by transmitting any required IR signals. And vice versa, the IR node might receive an IR signal and then command the IO node to toggle relays. Both could have access to the functionality of each other, but neither needs the libraries of the other.

The potential benefits are even greater than this. It would be simple to add a wifi node offering remote control of any of the IR or IO nodes capabilities without even needing to modify the existing IO and IR nodes. An SDcard node could be added offering a few additional commands that could allow any of the other nodes to use it for reading or writing to file. And a voice node might be added for making any announcements the other nodes commanded.

An interactive cluster of functionality could be easily built up to include anything and everything that anyone could wish an arduino to do, but split up across individual cluster nodes to avoid ever running out of resources. It allows easy modular development of quite complex and sophisticated systems, plus other benefits. It even offers integration of other platforms if wished, other than arduino - because a node is only a black box that can be instructed by plain text to action some inbuilt functionality and/or issue its own text commands for other nodes to action.

So I think there is good incentive for trying to do this, but it all depends on being able to find a practical way of having buffered variable-length strings for communications - the rest is just a relatively simple matter of parsing and appropriate conditional branching from a case statement.

but it all depends on being able to find a practical way of having buffered variable-length strings for communications

Is that many variable length strings in the one buffer?

If so then the technique for doing this is known as a linked list. What this does is to have one or two bytes at the start of the string that point to the end of the string. You need to write some house keeping functions like removing a string from a list by copying the end of the list over the string you want to remove.

Now you know the name you should be able to find numerous on line examples of this.

Have a look at the examples (especially the 3rd example) in Serial Input Basics and also the parse example. As written the examples have room for 32 chars - but that could easily be increased.

Have you the option to reduce the commands to a single character - for example R for “relay” - as that will making the parsing easier. For example <R,1,N,1230> would tell relay 1 to turn oN for 1230 millisecs

…R

I wish to be able to send sequences of plain text commands to an arduino to access its functionality (by conditionally branching to any internal functions with any appropriate parameters).

Probably not too difficult. First you may need to decide on the data format that is easiest for you to decode, then send the data in that format. Below is a simple method of packaging data that might be close to what you are looking for.

//zoomkat 11-12-13 String capture and parsing  
//from serial port input (via serial monitor)
//and print result out serial port
//copy test strings and use ctrl/v to paste in
//serial monitor if desired
// * is used as the data string delimiter
// , is used to delimit individual data 

String readString; //main captured String 
String angle; //data String
String fuel;
String speed1;
String altidude;

int ind1; // , locations
int ind2;
int ind3;
int ind4;

void setup() {
  Serial.begin(9600);
  Serial.println("serial delimit test 11-12-13"); // so I can keep track of what is loaded
}

void loop() {

  //expect a string like 90,low,15.6,125*
  //or 130,hi,7.2,389*

  if (Serial.available())  {
    char c = Serial.read();  //gets one byte from serial buffer
    if (c == '*') {
      //do stuff
      
      Serial.println();
      Serial.print("captured String is : "); 
      Serial.println(readString); //prints string to serial port out
      
      ind1 = readString.indexOf(',');  //finds location of first ,
      angle = readString.substring(0, ind1);   //captures first data String
      ind2 = readString.indexOf(',', ind1+1 );   //finds location of second ,
      fuel = readString.substring(ind1+1, ind2+1);   //captures second data String
      ind3 = readString.indexOf(',', ind2+1 );
      speed1 = readString.substring(ind2+1, ind3+1);
      ind4 = readString.indexOf(',', ind3+1 );
      altidude = readString.substring(ind3+1); //captures remain part of data after last ,

      Serial.print("angle = ");
      Serial.println(angle); 
      Serial.print("fuel = ");
      Serial.println(fuel);
      Serial.print("speed = ");
      Serial.println(speed1);
      Serial.print("altidude = ");
      Serial.println(altidude);
      Serial.println();
      Serial.println();
      
      readString=""; //clears variable for new input
      angle="";
      fuel="";
      speed1="";
      altidude="";
    }  
    else {     
      readString += c; //makes the string readString
    }
  }
}

I've been trying to digest Mikes idea of using linked lists, and trying to get my head around what would be needed... bearing in mind I'm not a programmer, so must use laymans logic.

First though, I seem to remember from somewhere that the serial buffer has an upper size limit (64 characters I think), so presumably my main priority must be to ensure I can keep pulling characters out of the serial buffer faster than it can be filled up, even though I may be too busy to do anything else with them at that time.

So my priority task is to keep on transferring characters from the serial buffer into a buffer of my own, which if I am understanding things correctly, could initially be an empty linked list with its start and end pointers pointing to the same address, and I assume that each character added would need to increment the end pointer.

But my main task is to look for the presence of a first-in delimiter in my linked-list buffer which I can remove along with any preceding characters, then use that delimited character string as appropriate. Which means I would then need to increment the start pointer according to how many characters were removed from the front of the linked-list.

I obviously can't just keep on incrementing start and end pointers indefinitely though, so presumably at some point I must make the end pointer wrap around to the vacated start, but as soon as I do, I think I would have pinned the tail to the nose and lost any ability for the linked-list to be able to expand if necessary (it makes me dizzy just thinking about this stuff).

The other thing I can't get my head around is how to keep track of an incoming delimiter. It would be a great time-waster to have to keep counting forwards from the current beginning of the list when looking for the next delimiter (especially if one hasn't even arrived yet) so I guess it would probably be better to spot a delimiter whenever one is actually pulled from the serial buffer, and then record its absolute position in my buffer for later use. Can't do it for one without doing it for all though, else some could slip through and be lost, so presumably I would need a separate FIFO buffer to stuff any delimiter pointers into until they could be dealt with.

So am I pointing in the right direction with this, or just getting hopelessly lost?

In order to answer your previous questions, and help make things clearer, I'll step back a bit from specifics to give a better overall picture by way of an example:-

Lets say we have some PIR sensors, some relays, and an IR sender - all on different arduino nodes, all able to send plain text strings to each other, and all parsing any incoming strings for any recognised command keywords (plus optional data) against an internal list of recognisable commands according to their own local functionality.

Commands could be anything that any of the nodes have been programmed to respond to, so can easily include macro functions for things like timers and conditional execution dependent on other things such as Night or Day.

Rather than hardcoding these triggers and responses to any of the individual nodes, it would be more convenient for these configurations to be at a centralised and easily maintainable location such as a config file on SD. So lets add an SD node, and make it parse incoming serial for a command we will call "Action". If the SD node finds a match for "Action" during parsing it will know it also needs to parse out an accompanying triggering parameter, eg: "PIR3", which will have a corresponding matching entry in the config file. The config entry could be something like this...

"PIR3 (front porch): TVmute, Porchlight ON".

The SD node would then "Action" the appropriate "PIR3" entry by parsing out any delimited command strings and sending any it doesn't recognise as an internal command on out to the other nodes via serial port. So the IR node would then receive and parse out "TVmute" and transmit the corresponding IR signal, and the relay node would parse out "Porchlight" plus its "ON" parameter and activate the corresponding relay.

That is just a simplistic example to help illustrate the interactive communication between nodes for shared use of all available cluster functionality. But it also shows the open-ended opportunity for enhancement and expansion. For instance we could easily add a "Pause" command to the SD node for invoking an internal non-blocking delay function, and a "NoRetrigger" timer for preventing any unwanted repeat trigger actioning (both these macro commands would be accompanied by a specified 'duration' parameter). Let's also add a new voice node that recognises the command "Announce" plus an accompanying parameter. Now, the updated config file entry could be something like this...

"PIR3 (front porch): NoRetrigger 3 mins, TVmute, Porchlight ON, Announce Visitor, Pause 3 mins, TVunmute, Porchlight OFF".

That's already on the way to becoming an intelligent and useful automation system.

Obviously every trigger input would be assigned its own corresponding list of responses in the config file, which could include any of the functionality that any of the nodes could respond to.

Adding new functionality to the cluster would be as simple as adding another command keyword to any nodes list of commands to be parsed for, which when matched would branch to an appropriate internal function. So adding a "WriteLOG" command to the SD node (and having it parse for an accompanying "Details" parameter) could provide a facility for every node to be able to record events to an event log just by sending the serial command "WriteLOG (details)".

Sophistication could quickly grow, but so too could string length, hence the need for handling potentially long strings of variable length, and also for appropriate buffering to prevent anything being lost during the bursts of chatter caused by any multiple event triggers.

Assuming things are possible, it offers a way to take advantage of much more functionality than could be achieved by any lone arduino or ESP.

So I hope that has helped answer any questions and filled in any blanks guys, and I hope the concept is thought worthy enough of some support to help turn into a reality.

Electroguard: So my priority task is to keep on transferring characters from the serial buffer into a buffer of my own

But my main task is to look for the presence of a first-in delimiter

The code in the third example inSerial Input Basics does these.

Commands could be anything

Rather than hardcoding these triggers and responses to any of the individual nodes, it would be more convenient for these configurations to be at a centralised and easily maintainable location such as a config file on SD. So lets add an SD node, and make it parse incoming serial for a command we will call "Action". If the SD node finds a match for "Action" during parsing it will know it also needs to parse out an accompanying triggering parameter, eg: "PIR3",

could be something like this...

"PIR3 (front porch): NoRetrigger 3 mins, TVmute, Porchlight ON, Announce Visitor, Pause 3 mins, TVunmute, Porchlight OFF".

My concern is that you will run out of memory space in your Arduino. It is all very well to have a large quantity of data on an SD Card - but each of the "actions" in the SD Card will need corresponding Arduino code and, generally speaking, storage for the code will use more bytes than its entry on the SD Card.

...R

The serial buffer is only 64 bytes by default, you can set it to be any size you want. So there is no need to pull data out and put it elsewhere just use the buffer and get data out of it when you can cope.

Wrapping round a circular buffer does not have any drastic effect like you think, the input and output pointers still work.

However if you have delimiters in your input data you don't need a linked list, you just read bytes until you find the delimiter.

Yes Robin, (I'm another Robin BTW) I take your point about memory quickly getting used up if everything parsed from the SD line entry needs corresponding matching code. But that shouldn't be a problem, because hardly anything read from a config entry would be recognised as a local command requiring a local response - most parsed config command strings would be locally un-recognised so simply squirted out the serial port for actioning by any other nodes which might be interested.

And just to set matters straight cos I've kept mentioning serial port, communications could actually be by any method that doesn't begrudge using the required resources for all nodes.

Serial would be fine for individual node development using serial monitor or terminal for interaction, and it could even be great for a simple 2 node system, but I was thinking RS485 might be the most practical form of cluster connection, allowing many nodes to be connected in parallel over relative long distances.

My fingers-crossed intention is to make cluster processing a realistic practical possibility for anyone else who would want to use it - I have to get it working first though, of course, and it's all completely new territory for me which is way out of my comfort zone.

But, Mike, you've given me great optimism that things might be as simple as just increasing Robins serial example3 buffer to 200? chrs and then grabbing out everything up to the next delimiter whenever possible. Wow, if it's that easy I won't know whether to laugh or cry. Thanks again, guys.

Electroguard: most parsed config command strings would be locally un-recognised so simply squirted out the serial port for actioning by any other nodes which might be interested.

I can see how that could work.

But, Mike, you've given me great optimism that things might be as simple as just increasing Robins serial example3 buffer to 200? chrs and then grabbing out everything up to the next delimiter whenever possible. Wow, if it's that easy I won't know whether to laugh or cry

Laugh, please. Or at least smile.

...R

Oh, I'm already smiling in anticipation Robin - but I'd like to be able to get a smile out of Mike also, cos someone calling themselves Grumpy sounds in need of being cheered up (unless they're friends with Snow White, of course).

Well that's wiped the smile off my face.

Ran some tests using Robins serial examples to confirm the serial buffer is indeed 64 bytes, then searched around to find out how to increase it.

This article claims to offer the easy way... "The solution is to create a complete copy of the arduino core code, modify the buffer size in the new core code and then to create a new board which is listed in the Arduino IDE which uses this new core directory."

That sure aint gonna be practical for every different type of arduino and ESP device used on anyones multiple node cluster, so unless there is a more practical later IDE method I haven't yet found, I guess that that's the concept finished before even getting started.

Anyway, thanks for your time guys, and sorry for wasting it.

Electroguard:
Ran some tests using Robins serial examples to confirm the serial buffer is indeed 64 bytes, then searched around to find out how to increase it.

With my code the 64 byte size of the buffer is irrelevant because the code keeps emptying the buffer before it overflows.

…R

Yes, thanks Robin. I had noticed that, but the problem is when the processor is busy doing other things, perhaps reading interupt-driven IR signals (or anything a bit time-consuming).

Any commands+parametersstring greater than 64 chrs that is sent by any other node before the local processor can return to pulling from the buffer again will end up being truncated and lost.

I'd already tried it with your example: I set numChars = 200, added a few seconds delay at the end of Setup (so that it was after Serial is initialised but before Loop starts pulling from the buffer), then I pasted a long string into the serial monitor and sent it during the delay period, and your sketch could of course only pull out the first 64 chrs that were in the buffer, everything after had been lost.

No, I'm afraid fully buffered long strings are crucial.

But I'm already trying to think of possible tangents: I'll check out software serial in case that can have a bigger software buffer more easily defined which might be able to be interupt driven so as not to miss anything. Also need to find out about the RS485 interfaces in case they have their own larger hardware buffer. Another possibility might be an interupt-driven buffer-emptying routine if it could be suitably triggered dependent on buffer contents. All new territory for me though, and I'm not expecting an easy time. So yeah, I'm down, but not out yet.

"The solution is to create a complete copy of the arduino core code, modify the buffer size in the new core code and then to create a new board which is listed in the Arduino IDE which uses this new core directory."

I am sure the last time I did it I just had to change one of the system variables.

Electroguard: Yes, thanks Robin. I had noticed that, but the problem is when the processor is busy doing other things, perhaps reading interupt-driven IR signals (or anything a bit time-consuming).

I suspect, then, that the program is poorly designed.

...R

Mike, from what I've been able to find out, it seems that the hardware serial buffer size is defined in a serial definition file which is buried down in with the hardware-specific core files, such as:

Arduino>hardware>arduino>avr>cores>arduino>HardwareSerial.h

It has a default size of 64 bytes stipulated, but that can be over-ridden by entering an earlier #define entry in the file such as:

define SERIAL_RX_BUFFER_SIZE 128

But this hardware dependent file gets pre-compiled before the IDE sketch, therefore the buffer value is already hard-wired before the sketch gets compiled so it has no way to then change it.

That's my understanding of it, anyway.

There is a July 2015 posting which says "... in the future, the IDE may allow sketches to provide defaults for these defines. Then each project will be able to have a custom setting for things like this".

But that's only a future maybe mentioned after giving instructions for modifying the serial.h core file which was necessary just a few months ago, and I don't think there's been a major IDE upgrade since (but I'll need to check).

I suspect, then, that the program is poorly designed

. Robin, I don't even have a program designed yet, and there's no point in even trying until -and unless - I can get the crucial chassis in place to build on, and that's why I need to have bufffered long-string functionality before I can even consider going any further. I may not be a programmer, but I've learned enough from painful experience to know that if something can go wrong, it will - so it would be folly to blindly push ahead with something I know is already a problem just waiting to happen. And especially when I suspected this was going to be the key problem from the very beginning.

The other key thing I've learned is that arduino basically means compromise. When I say I don't have the program yet, I've actually got 3/4 of it which I needed to call a halt to when I ran out of arduino resources - which is why I've been trying to do this cluster processing idea in the first place.

I've got a very old computerised security/cctv system I am very dependant on which I created using Homeseer home-automation software and Homevision hardware (for IO etc) many years ago, but which is all now no longer available. After several scares, and becoming newly aware of seeing the previously unheard of arduino being mentioned everywhere, I thought it worth having a try to pre-empt disaster by seeing if I could create my own purely hardware arduino equivalent as a replacement.

I got a lot further than expected, but then ran out of resources - but by then my little Nano was monitoring 64 channel interupt driven inputs (PIRs, beam-breaks, magnetic contacts etc), all with voice announcements, selectable in 4 different languages, from a voice menu navigated by IR controller, every trigger event had an associated non-retigger timer, and events could do all the normal relay stuff, but also do things like send delayed sequences of IR commands for controlling the cctvs etc. So I was quite chuffed with my one and only arduino project, even if the software might have been an embarrassment.

The only way to get adequate hardware responses was to have IO interupts interupting IR interupts interupting timer interupts interupting voice interupts interupting IO interupts... so I can pretty much guarantee there can be no guarantee of being able to prevent the serial buffer overflowing unless it could also be interupt-driven. Or by thinning down the workload by distributing it across multiple nodes... requiring the catch-22 situation of dependable buffered long strings.

Running out of memory prevented me from completing things like SD centralised configuration files and event logs and storage of IR commands etc, or X-10 fire-cracker control, and web control etc. It can certainly all be done - even by me - but it can't all be done on a single arduino - not by anyone - cos there's simply not enough memory for all the libraries, interupts for responsive hardware, or processor time to cope with it all.

I'm fairly confident I could do it with 2 though - especially if one is an ESP8266 - but I haven't found a satisfactory way for linking them to allow them both complete freedom to communicate everything possibly required of each other to each other while still retaining their independence.

I'm hopeful I could do that with buffered long strings as I've already explained, and at least I know they are possible now, even though a bit messy. But that makes a difference, because it's easier to keep battling against currents if others are depending on you, but there's far less incentive just for something for yourself that may never be needed anyway.

So I'm sorry about the ramble, but I was feeling a bit like Dopey being ticked off by little miss perfect Snow White for not being able to reach the higher shelves that she can reach... and it aint really my fault if I was born an ugly short-ass, is it. Hey, I'm smiling again!

Electroguard: there's no point in even trying until -and unless - I can get the crucial chassis in place to build on, and that's why I need to have bufffered long-string functionality

That's exactly what my code is for. Try it. If it does not meet your need post the program and the data that demonstrates that and I will try to help.

...R

and I don't think there's been a major IDE upgrade since

Been at least two.