Project structure guidance

Hi everyone. My two previous posts on this forum (#1, #2) are related to a project I'm working on that involves controlling a variety of actuators using an Arduino Mega, which is triggered by commands sent serially from a Raspberry Pi.

It was clear from the responses to #2 that my approach is flawed. I'm taking a step back and reevaluating the structure of the project.


Background:
The Arduino will be responsible for the control of a range of actuators, including linear actuators and motors. The motors have their own controllers that I can communicate through a serial connection, and the linear actuators are controlled with relays -- but the actuator control isn't the main focus of this post.


Multitasking:
In post #1 I inquired about how to best handle tasks psuedo-simultaneously and the best advice I received was in post 6 - bascially to create a state machine for each actuator and have the main loop iterate over the state machines, changing their behavior based on whether they have achieved their target positions / speeds / etc. This process of checking and updating would happen quickly enough to achieve good-enough multitasking functionality.

My current plan is to represent each actuator with a struct like this:

struct actuator {
  char* name;
  unsigned int timer; // Current timer count
  unsigned int timeout; // Timeout set by command
  int current_position; // Actual position read from encoder
  int target_position; // Target position to try and reach
  unsigned int state; // To set the required behavior of the actuator
  }

Communication:
I'd like to be able to send packets of commands of a set structure (see below) from a Raspberry Pi to the Arduino over serial connection. A packet can contain one or more command, with no more than one command for each actuator. The Arduino will ingest and parse each packet for individual commands. I'm not sure what the most efficient way to handle the next step is -- but basically I need to iterate over the individual commands making the necessary changes to the relevant and then (or in the same for loop?) check the status of the actuators that weren't referenced in any of the commands to make sure they're on track.

This is how I envision the packet structure:

&XXYYZZZZ!

Where:

  • & signals the start of a new command
  • XX is the ID of the target actuator (i.e. 01 for motor_1)
  • YY is the ID of the command type (i.e. 03 for rotate_to_angle(arg))
  • ZZZZ is the argument arg from the previous bullet
  • ! signals the end of a packet, so it's possible to string together multiple commands like this:
&XXYYZZZZ&XXYYZZZZ!

My current approach:
This is a minimal (but compilable! as per @david_2018's advice in post #2) example of how I'm receiving the packets of serial data (note - this is without parsing the packet for individual commands):

// Max length of packet is max number of actuators - let's say 5 for this example
const unsigned int MAX_PACKET_LENGTH = 5;

// This flag lets us know when a full packet has been received
int PACKET_READY_FLAG = 0;

void setup() {
  // Open serial connection
  Serial.begin(9600);
}

void loop() {
  // Get incoming packet
  char *packet = get_bytes();

  if (PACKET_READY_FLAG == 1) {
    Serial.println("New packet ready.");
    }
}


char *get_bytes() {
  // This runs as long as there are available bytes in the serial buffer
  while (Serial.available() > 0) {
    // This is the position in the packet being written to
    static unsigned int packet_position = 0;

    // Here we initialize the packet (character array)
    static char packet[MAX_PACKET_LENGTH] = {0};

    // Read next incoming byte
    char in_byte = Serial.read();

    // No error handling here yet - this writes to and increments the position in the packet
    if (packet_position < MAX_PACKET_LENGTH) {
      packet[packet_position] = in_byte;
      packet_position++;
      }

    // This triggers the end of a packet
    if (packet[packet_position - 1] == '!') {
      PACKET_READY_FLAG = 1;
      return packet;
      }
    }
  }

Please note that packet is not meant to be written to / adjusted from outside the function get_bytes(), but will be read by other functions. This is mainly in response to this post of @J-M-L's (thank you for the input J-M-L!) As far as I know this presents a healthy use of static variables.


With that said hopefully my post #2 is now more clear. I need to parse packet for individual commands. My plan is to iterate over packet while copying the contents to a 2D "array of arrays", where each sub-array is an individual command. This is the function I wrote but I think there are a number of problems with it (you can paste this after the code I wrote above and it will still compile):

// The length of a command is 9 including "&" at the start
const unsigned int MAX_COMMAND_LENGTH = 9;

// This function parses the packet in static memory for individual commands
char *parse_commands(char *packet) {

  // Set up 2D array of commands
  static char commands[MAX_PACKET_LENGTH][MAX_COMMAND_LENGTH];
  static unsigned int commands_position = 0;

  // Iterate over packet in static memory and
  // look for "&" for the start of a new command, incrementing
  // commands_position when each new command is discovered
  for (int i=0; i<MAX_PACKET_LENGTH; i++) {
    if (packet[i] == '&') {

      // Write to command sub-array
      for (int j=i; j<MAX_COMMAND_LENGTH; j++) {

        // Is this the right way to copy a single character?
        commands[commands_position][j] = packet[j];
        }

      // Increment command array position
      commands_position++;
      }

      // This trips when end of packet is reached
      else if (packet[i] == '!') {

        // ---> Return
        return *commands;
        }
    }
  }

Bonus imaginary points for whoever can answer this:
Regarding "---> Return", I originally tried to simply put "return commands;" without * like I did with packets in get_bytes(), but it didn't compile (received the error cannot convert 'char (*)[9]' to 'char*' in return). I don't understand why. Shouldn't it simply return a pointer to the address of the first item in the 2D commands array?


That's enough for now.
To wrap up, my questions are:

  1. How can I improve my overall strategy?
  2. How can I improve get_bytes()?
  3. Am I on track with parse_commands()? What's a more efficient (but still readable) way to parse packet?

Thank you! I'm learning a lot from this forum.


Edit 1: Added error received when trying to compile without * in parse_commands().

If you haven't turned up compiler warnings to the max yet, I suggest that you do. I would expect that get_bytes would be giving a warning about not all code paths have a return value.

You have too many detailed implementation questions to answer in advance of project construction. You need to break down tasks so you can develop and test them individually. This would allow you to complete some modules and stop thinking about them so you can get on to some other problems.

You said that you, "take a step back" but in some places here you've leaped right into detailed code specification that has no obvious connection to the overall task. For example the "Multitasking" where you immediately dig into a data structure format. In that place, should be an explication of the state machine states, what they will do, and maybe how many state machines as sometimes you need more than one...

Disregarding all that, leaves a question about a specific section of code, the parser. But you are vague about the problems with it, you should know by now that we would need more details about the specific "number of problems" that you say it exhibits. Also some explanation of how it should work, since there is no documentation and only minimal internal comments. When you say something like "it didn't compile" it is not helpful, you should post the complete error listing and you should also know that by now.

To address one of your points, strtok() is usually the tool for parsing text.

A high level question - why do you place the packets in an array? Can they not be processed individually, as they are received?

Summarizing, you're asking for advice on your project structure, but not saying in any detail what that is. What you have done, is say some things about what it should do, but in very vague language that invites simple assent, like "sounds good, try it!".

Thanks, good advice. I will do this.

Thanks for your response. Hopefully it's clear that I'm doing my best to format my post in a way that respects those who will take the time to read and answer it - but there's always room to improve and responses like yours help.

in some places here you've leaped right into detailed code specification that has no obvious connection to the overall task.

For a beginner like myself, code specification and the overall structure of the project go hand-in-hand. The specific code related questions I asked are meant to help me determine whether my approach even makes sense.

Also some explanation of how it should work, since there is no documentation

Fair enough. I'll add that.

When you say something like "it didn't compile" it is not helpful, you should post the complete error listing and you should also know that by now.

I'll add the error messages.

A high level question - why do you place the packets in an array? Can they not be processed individually, as they are received?

To me it makes sense to ingest the entire packet and confirm that it is valid before trying to process it. Maybe there's a way to perform these checks as the packets are received. I'll give this some thought.

But that is only one packet isn't it? Maybe you are placing the commands in an array? So then my question would be why put those in an array and not just execute them as you parse....

On the point of receiving the entire packet, how are you validating it?

I think you should try to make some of your functions work independently first before you try to put everything together. So for example, break the actuator control code and the packet input code into two projects, complete each one and test it separately.

You are in control of the packet format?

If so, I would suggest something with field separators

&X,Y,Z!

With one command per packet. Then you can use Serial.readBytesUntil() and Serial.parseInt() and keep things much simpler than the code you suggested above. No need for buffers and tokenising.

But that is only one packet isn't it? Maybe you are placing the commands in an array? So then my question would be why put those in an array and not just execute them as you parse....

Executing while parsing could work, but I'm trying to maintain modularity and to me it seemed to make sense to separate the parsing and execution. But I'll think about what you're suggesting, thanks.

On the point of receiving the entire packet, how are you validating it?

The code I posted assumed that a valid packet was sent. It didn't seem relevant to the goal of the post to include error checking, etc.

I think you should try to make some of your functions work independently first before you try to put everything together.

Yep, that's what I'm doing. None of the code responsible for actuator control appears here and I'm working on that separately.

I'm in control of the packet format, yes. Great advice! I'll look into these. Thanks.

Packet format, why? Nice word, "packet" but what you intuitively want is just a command-driven sketch that can take serial from the rPi.

The most common example is GPS parsing of NMEA sentences.

Years ago, I wrote a silly example that takes the concept into a working example:
https://forum.arduino.cc/t/dont-cross-the-streams-fp-scientific-calculator-serial-co-processor/144054

There is no sense making this any harder than it needs to be. A nice feature would be to write a handshake routine so that the Arduino could ask for a resend if line parsing failed.

Ray

Thanks for the reply - but I am not sure I follow you.

Packet format, why?

Why not?

Nice word, "packet" but what you intuitively want is just a command-driven sketch that can take serial from the rPi.

What's wrong with the word "packet"? I don't understand the distinction you're making.

Years ago, I wrote a silly example that takes the concept into a working example:

This is helpful. Thanks.

There is no sense making this any harder than it needs to be.

Well that's obvious :wink:

A nice feature would be to write a handshake routine so that the Arduino could ask for a resend if line parsing failed.

I plan on adding this. Thanks!

It’s OK but I would say it’s not great. You keep a pointer to the static buffer outside the function but if the function starts receiving data again, the pointer you have no longer points to the old data.

Having just a global bufffer would make everything clear and you would not need to return any pointer etc…

You might look to Gammon Forum : Electronics : Microprocessors : How to process incoming serial data without blocking and the the g-code parsing in 3d printer software, such as Teacup_Firmware/gcode_parse.c at 9b934c364e52c976adaa08fabdffd1f99d6af3eb · Traumflug/Teacup_Firmware · GitHub -- it parses characters as they come in, and stages data into a set of registers, and when the line is complete, sends it for processing.

With Gammon's code, an 8 digit command could fit easily into a 4-byte unsigned long that could be checked and separated into ID, command, and arg with simple integer arithmetic.

I grew up with post-fix code like RPM and Forth, so I'd think of your & and ! as commands that would tell the code to act on the prior numbers.

Are the actuators all independent? Reason I ask is that it seems your command language does not have a way to set sync points, or check for limit switches etc. How will you do homing for example?

Is that the only media? Will there ever be a radio, ethernet, or some other connection between the two?

class?

Given the fact that you've defined a prefix (&) and a suffix (!), what problem is solved by stringing commands?

That sounds like unbounded data. Is it?

Fair enough. Thanks!

Funnily enough we just got a 3D printer as well! Thanks for the response and material.

They are all independent, yes. The actuators for which homing / limit switches are relevant have their own control units (ODrive motor controllers) with which I'll be able to communicate and close the control loop.

Is that the only media?

All communication will take place over serial connection.

class ?

To my understanding the only difference is public / private variables and methods. At this stage I don't see a benefit to using private variables and methods but I very well may in the future.

Given the fact that you've defined a prefix (&) and a suffix (!), what problem is solved by stringing commands?

This is a good question. In my mind, I want things to be executed as quickly in succession as possible, so I imagined the rPi sending a bunch of commands at once which the system will parse and execute. But given that most of the actuators are independant in their control and functionality the rPi will in any case be relating to each one individually - so it's probably faster to just send a command as soon as it's ready instead of aggregating them.

That sounds like unbounded data. Is it?

I had to look this up - I'm still not entirely sure what this means in this context. Could you help me understand better?

Publishing an API is some sort of commitment that it’s going stable for years.

if the code is only for you then it’s not a big issue. When you start sharing code and other people use it, whatever is public can be used which might prevent you from further optimizations or changes later on if you want to maintain backward compatibility.