Organising code for a large-scale, modular project

[proffalken]

Hey folks,

I've got a working prototype of a project for a model railway that I'm writing, and now I want to make it more organised.

I'm a python dev, so I'm used to organising my code in classes etc, and I'm now looking to turn my arduino project towards a similar approach.

The project involves a number of physical "modules" (ESP8266-based in my case) that connect via MQTT over WiFi and flash LED's, operate servos, or switch relays

I'm writing a library to bundle the wifi/MQTT connections into a reusable space, however I'm now thinking that as all the modules will be based on the same chip is it worth bundling the various classes and functions that will drive the motors/servos into the same library or a different one?

As an example, I'll have a number of signals on my layout controlled by servos. There will also be "animations" on the layout from LED's (fires flickering in houses etc).

At the moment, the code looks a bit like this:

rail_control.ino
  |_ #include "controller.h"
      |_ WiFiSetupAndControl(ssid, password)
      |_ MQTTSetupAndControl(esp_client)
      |_ SignalControl(signal_state)
      |_ LEDControl(pattern, delay_between_state_changes)

I'm wondering if it's worth splitting SignalControl and LEDControl out into their own separate libraries, or at least into dedicated files given that "controller.h" is now well over 500 lines long, but I'm not sure how I'd best do that in this language!

Any help appreciated, especially if it's links to example projects where this has already been done.

I've looked at platform.io, and I'm still not ruling it out, but it does seem overkill when I'm going to be deploying to the same hardware platform for all the devices.

[6v6gt]

This is a general area relevevant to large C++ developments and I hope you get some good answers.
You've also illustrated it with some specific examples so I address those.

You appear to have a single header file which is used by very varied functions. For example WiFiSetupAndControl() and LEDControl(). More usual is that these would each have their own program (.cpp) and header file (.h).

A class model is useful but C++ also has a namespace model for encapsulation. For the case where you define a class and only ever create one instance of it, a namespace is a valid alternative to a class construct. It is the "why create a cookie cutter and make only one cookie" argument. For Arduino programming, I use a namespace model which someone ( I forget who ) published here on this site. That is each namespace has its own setup() and loop() method and this can make a very clean sketch. Where you need to create multiple instances of something, say buttons on a touch screen or queues etc., then you really need a class construct.

[Robin2]

proffalken:
I'm writing a library to bundle the wifi/MQTT connections into a reusable space, however I'm now thinking that as all the modules will be based on the same chip is it worth bundling the various classes and functions that will drive the motors/servos into the same library or a different one?

A few thoughts

I have never bothered to create a formal Arduino library - I don't see any advantage unless it is for the convenience of other programmers.

I suggest separating your ideas about classes from your ideas about separate files. My approach had been to put a certain category of functionality into .h file (I'm a reluctant C++ programmer, I never bother with .cpp files, and the compiler doesn't care). For example for my model train wireless control project I have files called trainWirelessControl.h and turnoutControl.h. I also have a file called dataStructs.h as I found it easier to have all of them in one place because some of the data elements are common.

Quite separately, I would be very interested if you could provide an overview of the functionality of your project and its communication system. My goal is to have a layout that is entirely automated with the primary control in the hands of a Python program. My Arduino Mega just deals with the hardware (turnouts and LDRs) and nRF24 wireless communication with the trains based on instructions from Python.

...R

[gfvalvo]

6v6gt:
A class model is useful but C++ also has a namespace model for encapsulation.

@6v6gt: Could you post a skeletal example of this technique? I've often run into the situation where I define a class and then only instantiate one object of that class. To put your metaphor another way -- you shouldn't have to build a bridge just to cross a stream once.

[gfvalvo]

So, there are lots of opinions and techniques regarding large project organization.

My personal preference is the method used by nearly all Arduino libraries as well as by professional programmers working on large software projects in industry.

I break the program's functionality into logical modules that ideally can be developed and tested stand-alone / individually -- even by different people.

Each module has a .h file and a .cpp file. The .h file contains the module's interface -- i.e. only the information that other modules need to use this module's public functionality. After the #include guards, this typically means:

• Prototypes for public functions.
• Class declarations.
• 'extern' declarations of global variables.

The .h file may also contain class method implementations within the class declaration - but only if said implementations are short (couple of lines). It should also contain #include directives of any .h files needed by the contained declarations or by modules that use the interface. If there are any .h files only required by the implementations, then they should NOT be #included in the .h file but in the .cpp file.

The .cpp file should contain:

• #include of the associated .h file.
• #includes of any .h files required by the implementations but not the interface.
• Function implementations.
• Class method implementations.
• Class static variable definitions.
• Definitions of global variables declared as 'extern' in the associated .h file.

Not only does this method provide superior modularity, but it allows you to take advantage of file-level scope control using the 'static' directive. This lets you keep functions and global variables private to the implementation file if other modules have no need for them.

Using this method there should only be ONE .ino file -- the main one that contains setup() and loop().

[6v6gt]

gfvalvo:
@6v6gt: Could you post a skeletal example of this technique? I’ve often run into the situation where I define a class and then only instantiate one object of that class. To put your metaphor another way – you shouldn’t have to build a bridge just to cross a stream once.

A relatively large project I published here (Speaking Clock) uses this namespace model:

Arduino ESP8266 Speaking Clock

Here is a sample extract:

// ******************************************************
// file SpeechEngine.h
// ******************************************************
#ifndef _SpeechEngine_h
#define _SpeechEngine_h

namespace nsSpeechEngine {
. . .
struct track_t {
  // start and end of each track in units of 10 mS
  uint16_t st ;
  uint16_t end ;
  uint16_t duration ;
} ;
. . .
. . .

// prototypes

void workflow_AT_RFZ_24H() ;

void setup() ;
void loop() ;

} // namespace nsSpeechEngine
#endif



// ******************************************************
// file SpeechEngine.cpp
// ******************************************************
#include "SpeechEngine.h"

namespace nsSpeechEngine {

void workflow_AT_RFZ_24H() {
  . . .
}

void  setup() {
  // initialise SpeechEngine
  . . .
}

void loop() {
  . . .
}

} // namespace nsSpeechEngine


// ******************************************************
// file .ino file
// ******************************************************
#include "SpeechEngine.h"

void ICACHE_FLASH_ATTR setup() {
  Serial.begin( 115200 ) ;
  . . .
  nsSpeechEngine::setup() ;
  . . .
}

void loop() {
  . . .
  nsSpeechEngine::loop() ;
  . . .
}

[proffalken]

Thanks all, this is really valuable so far.

gfvalvo:
My personal preference is the method used by nearly all Arduino libraries as well as by professional programmers working on large software projects in industry.

I break the program's functionality into logical modules that ideally can be developed and tested stand-alone / individually -- even by different people.

..........

Using this method there should only be ONE .ino file -- the main one that contains setup() and loop().

This is the approach I envisaged taking with my code, it's interesting to see the other options though.

Out of interest, how do you test your code? Do you have unit-tests etc? If so, which frameworks are you using?

Thanks again for the responses so far!

[proffalken]

Robin2:
Quite separately, I would be very interested if you could provide an overview of the functionality of your project and its communication system. My goal is to have a layout that is entirely automated with the primary control in the hands of a Python program. My Arduino Mega just deals with the hardware (turnouts and LDRs) and nRF24 wireless communication with the trains based on instructions from Python.

...R

Thanks, I'll see what I can get working, and then I'll post it for sure.

Similar to you, this is using some python as the controller, although I'm using DCC for the train control because it's N-gauge so I'd struggle to fit the processors inside the locos!

[gfvalvo]

proffalken:
Out of interest, how do you test your code? Do you have unit-tests etc? If so, which frameworks are you using?

I use Eclipse / Sloeber for all my serious development work. Since I'm the only one writing code for my projects, I simply test as I go, sometimes creating simple temporary projects to test various modules.

[proffalken]

gfvalvo:
I use Eclipse / Sloeber for all my serious development work. Since I'm the only one writing code for my projects, I simply test as I go, sometimes creating simple temporary projects to test various modules.

Fair enough, I've spent the last 10 years as a "DevOps Consultant" and I'm trying to bring automated testing into projects I do outside work as well - I'm tempted to do the same with this code just to see if I can!

[wildbill]

Most Arduino projects, and especially the ones you see here where people are asking for help, tend to be very small. Unit tests are probably overkill. However, just like source code control, they can prove their worth even on a tiny project.

So go for it - I'd be interested in how it goes.

[proffalken]

wildbill:
Most Arduino projects, and especially the ones you see here where people are asking for help, tend to be very small. Unit tests are probably overkill. However, just like source code control, they can prove their worth even on a tiny project.

So go for it - I'd be interested in how it goes.

Yeah, in 10 years of mucking about with this kind of stuff, this is the first project that's made me think "I could do with my own library here...", and from there everything's just spiralled into a "proper" project like I'd do at work!

Otherwise, I'd just write the .ino and test it by looking at the output on the serial/physical ports/pins...

[Robin2]

proffalken:
Thanks, I’ll see what I can get working, and then I’ll post it for sure.

I am very interested to get a brief description of what you are planning to create.

Similar to you, this is using some python as the controller, although I’m using DCC for the train control because it’s N-gauge so I’d struggle to fit the processors inside the locos!

I have a Bronze award from the N Gauge Society (in 2013) for an N-Gauge Farish large prairie that I converted to battery powered radio control using Deltang wireless gear that I re-programmed with the Arduino IDE. I also converted a couple of other locos but I have given up on N-Gauge because I find it visually too small - the trains seem too far away. I am working with 009 now as my space for a layout is too limited for 00 Gauge or H0.

IMHO battery power makes most sense for the small gauges because they are the ones with the worst power pick up problems,

…R

[gfvalvo]

6v6gt:
Here is a sample extract:

// ******************************************************

// file SpeechEngine.h
// ******************************************************
#ifndef _SpeechEngine_h
#define _SpeechEngine_h

namespace nsSpeechEngine {
. . .
struct track_t {
 // start and end of each track in units of 10 mS
 uint16_t st ;
 uint16_t end ;
 uint16_t duration ;
} ;
. . .
. . .

// prototypes

void workflow_AT_RFZ_24H() ;

void setup() ;
void loop() ;

} // namespace nsSpeechEngine
#endif

// ******************************************************
// file SpeechEngine.cpp
// ******************************************************
#include "SpeechEngine.h"

namespace nsSpeechEngine {

void workflow_AT_RFZ_24H() {
 . . .
}

void  setup() {
 // initialise SpeechEngine
 . . .
}

void loop() {
 . . .
}

} // namespace nsSpeechEngine

// ******************************************************
// file .ino file
// ******************************************************
#include "SpeechEngine.h"

void ICACHE_FLASH_ATTR setup() {
 Serial.begin( 115200 ) ;
 . . .
 nsSpeechEngine::setup() ;
 . . .
}

void loop() {
 . . .
 nsSpeechEngine::loop() ;
 . . .
}

Thanks @6v6gt. Looks like this technique gives you similar functionality to defining a class with only static methods / variables and no instantiated objects.