This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Developing Control Applications

Developing Control Applications

MOTORCORTEX provides out of the box solution to develop Hard-Realtime Control Applications in C++ with an integrated high-speed and secured communication for distributed systems. It supports popular industrial buses, like EtherCAT.

The Parameter Tree

In a Motorcortex application all Modules and variables are organized in a tree structure; the Parameter Tree. The Parameter Tree is used to communicate data between tasks and to the outside world in a thread-safe manner. In the tree the modules are represented as subtrees (folder) and variables are represented as leafs.

The Parameter Tree contains a snapshot of all the registered inputs, outputs and internal data of the modules at the current time. Each Task or Module can create its own subtrees (folder) in the Parameter Tree. Modules can be nested and register their own parameters in the Tree.

The Parameter Tree always starts from the root node, which is passed to the modules and tasks that then can register their own parameters or create submodules that register their own data.

The Communication Server

The Communication Server provided by Motorcortex has two parts:

  1. the first part is a Signaling Server which uses a Request/Reply reliable messaging pattern.
  2. The second part is a Publishing Server or Publisher which uses a Publish/Subscribe best-effort delivery messagin pattern.

A Signaling Server is required for Remote Procedure Calls and for managing the control application, for example updating parameter values. It is also responsible for managing the Publishing Server.

The Publishing Server can organize data into groups and publish them with the requested frequency. The Publisher is used to send Real-time data continuously.

The Signaling Server is set up in the main(…) function and runs until a terminating signal is received.

Because the Publisher has a cyclic behavior, it is added to a Task that executes it with the given update rate. An example how the publisher is created is shown below:

1 - Setting-up your Developement Environment

The main tool of use for building, testing and deploying software is CMake Cmake is widely accepted and supported by many IDEs, which means that the user is free to choose his/her own development environment.

To make things more convenient, there are Motorcortex plugins for CLion (https://www.jetbrains.com/clion). CLion is an advanced IDE (Integrated Development Environment) that has many tools that makes programming easy, is available across platforms and is capable to cross-compile and deploy Motorcortex applications straight to the controller with a click of a button. This section is a guide, how to install CLion and the Motorcortex plugins from the CLion Marketplace and get started quickly with developing Motorcortex applications.

Setting up your development environment for creating control applications is different for every machine. Please select the operating system you are u using:

Linux Windows MacOS

Install Build Tools

Before Installing Clion it is important to install build-essential tools

   sudo apt-get install build-essential

Installing CLion For Linux

Installing CLion has be done by downloading Clion directly from the Jetbrains website Download CLion from: www.jetbrains.com/clion/download

How to Install CLion is explained in their documentation: www.jetbrains.com/help/clion/installation-guide.html#standalone

To use Clion you will have to create a Jetbrains account and start with a free trail.

Installing Motorcortex Templates Plugin (for clion)

After installing Clion the first step is to install a the Motorcortex Templates plugin. This plugin will provide you the option to create a new Motorcortex project.

  1. In the Main Menu on the top left corner open: File → Settings, then find the Plugins menu.

  2. In the plugins menu search for Motorcortex.

  1. Install the Motorcortex Templates plugin and Restart IDE.

After installing the Motorcortex templates and restarting the Clion, you can create new Motorcortex projects in the main menu.

  1. In the main menu open: File → New Project, in the pop-up under C++ choose C++ Motorcortex select the location where you want to save your project and press the Create.

  1. Congratulations now you can start developing your Motorcotex application.

The Motorcortex development toolchain

There are two options when developing Motorcortex applications:

  • Use remote development and compile the application on the Motorcortex Target System;
  • Cross-compile on your development PC and deploy the executable to the Motorcortex Target System

The first method is easy to set up, it does not require additional dependencies however the downside is that a compile time for the large projects may be significant. The second method overcomes this problem, but requires a cross-compilation tool-chain to be set up.

Set-up Cross Compilation (In Clion)

For cross compilation you use … (with remote device libraries).

Install Motorcortex toolchain on your system

For Cross compilation you will need a Motorcortex SDK you can download the SDK (Software Developement Kit) from the motorcortex.io store by pressing the download button below.

  1. Extract the archive and place the .sh file in a folder of your preference.

  2. In the terminal browse to the folder and run the .sh file from the terminal, for example:

   sh ./motorcortex-glibc-x86_64-motorcorcortex-image-dev-corei7-64-toolchain-X.X.X.sh
  1. You will be asked to enter the target directory where to install the SDK to. It is a good idea to keep the SDK in the home folder, where administrative rights are not required for example:
   ~/mcx-sdk

Mounting Sysroot on your system

The first step to Cross Compilation is to mount a Sysroot directory to your system that is used for cross-compilation. A `Sysroot is a directory which is considered to be the root directory for the purpose of locating headers and libraries of your application.

  1. Generate a new key pair with the following command.
   ssh-keygen -t rsa

  1. Use ssh-copy-id to provide a temporary access without a password to the remote Motorcortex host. If you are asked to fill in a password use vectioneer.
   ssh-copy-id admin@192.168.2.100

  1. Create a folder to mount the remote Motorcortex file system on with the following command.
   mkdir ~/mcx-sysroot
  1. Ensure that sshfs is installed with the following command:
   sudo apt-get install sshfs
  1. Mount the Motorcortex file system with the following command:
   sshfs admin@192.168.2.100:/ ~/mcx-sysroot/

Configure toolchain in clion.

Configure the compiler paths to use the Motorcortex cross-compilation toolchain.

  1. Open File → Settings → Build Execution Deployment → Toolchains.

  2. Create a new System Toolchain with the name motorcortex.

  3. Fill in Make, C Compiler and C++ Compiler paths to point the Motorcortex SDK as shown below.

  • Make:
   /home/ubuntu/mcx-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/make
  • C Compiler:
   /home/ubuntu/mcx-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc
  • C++ Compiler
   /home/ubuntu/mcx-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-poky-linux/x86_64-poky-linux-g++

  1. Press Apply to save the changes.

Setup toolchain plugin

Configure sysroot path to use the headers and libraries from the remote host.

  1. Open File → Settings → Build, Execution, Deployment → **Motorcortex SDK and fill in the Sysroot path path the mounted folder (mcx-sysroot).
   home/ubuntu/mcx-sysroot

Add deployment connection to remote host (IS THIS STILL NEEDED IN THE NEW VERSION ALEXEY!!??)
  1. Add the connection to the remote Motorcortex host. Open Tools → Deployment → Configuration, add a new SFTP connection named mcx-host.

  2. select or add ssh configuration.

Press the Test Connection button to make sure that the filled data is correct.

Press the Autodetect button to set the root path. Uncheck the Visible only for this project checkbox make this host visible in all the projects.

press ok

Cross-compiling Motorcortex applications.

edit configurations.

select mcx dingie

save password checkbox

name cross-mcx-example

deployment profile to mxc-host

press ok

  1. In CLion open the project configuration menu.

  1. Click on + and select Motorcortex Remote Application. In the deployment profile field select Motorcortex remote host.

copy config folder

  1. In the Run/Debug configuration select the just created cross-compiled application with CrossDebug/CrossRelease build profile.

  2. Press the hammer button to cross-compile, play to copy and run on the remote Motorcortex host and the bug button to start a remote debug session.

Note if CMake CrossDebug/CrossRelease don’t work remove the “Motorcortex Remote Application” from Run/Debug Configurations and try again

2 - Setup Development Environment

The main tool of use for building, testing and deploying software is CMake Cmake is widely accepted and supported by many IDEs, which means that the user is free to choose his/her own development environment.

To make things more convenient, there are Motorcortex plugins for CLion (https://www.jetbrains.com/clion). CLion is an advanced IDE (Integrated Development Environment) with many tools that make programming a Motorcortex application more easy, this IDE is available across platforms and is capable to cross-compile and deploy Motorcortex applications straight to the controller with a click of a button. This guide will show you how to install CLion, Motorcortex plugins from the CLion Marketplace and get started with developing Motorcortex applications.

The Motorcortex development toolchain

There are two options when developing Motorcortex applications:

  • Use remote development and compile the application on the Motorcortex Target System;
  • Cross-compile on your development PC and deploy the executable to the Motorcortex Target System

The first method is easy to set up, it does not require additional dependencies however the downside is that a compile time for the large projects may be significant. The second method overcomes this problem, but requires a cross-compilation tool-chain to be set up.

2.1 - Install Development Environment

Installing your development environment for creating control applications is different for every PC. Please select the operating system you are u using:

Linux Windows MacOS

Install Build Tools

Before Installing Clion it is important to install build-essential tools

   sudo apt-get install build-essential

Installing CLion For Linux

Installing CLion has be done by downloading Clion directly from the Jetbrains website Download CLion from: www.jetbrains.com/clion/download

How to Install CLion is explained in their documentation: www.jetbrains.com/help/clion/installation-guide.html#standalone

To use Clion you will have to create a Jetbrains account and start with a free trail.

Installing Motorcortex Templates Plugin (for clion)

After installing Clion the first step is to install a the Motorcortex Templates plugin. This plugin will provide you the option to create a new Motorcortex project.

  1. In the Main Menu on the top left corner open: File → Settings, then find the Plugins menu.

  2. In the plugins menu search for Motorcortex.

  1. Install the Motorcortex Templates plugin and Restart IDE.

After installing the Motorcortex templates and restarting the Clion, you can create new Motorcortex projects in the main menu.

  1. In the main menu open: File → New Project, in the pop-up under C++ choose C++ Motorcortex select the location where you want to save your project and press the Create.

  1. Congratulations now you can start developing your Motorcotex application.

2.2 - Setup Remote Development

Set-up Remote Development (In CLion)

The most simple way to develope control applications is to use remote development. With remote development you compile the application and put it on your controller.

Check Connection

Connect your controller to your PC and make sure you have a connection. If you don’t know to connect to a Motorcortex controller make sure to check out Connect your PC.

  1. Before we start we have to make sure our connection works. open the Terminal located on the bottom of your IDE or press Alt+F12.

  1. In the terminal ping to your controller with the following command.
   ping 192.168.2.100 -c 3
  1. The terminal should look something similar to this.

If you are unable to ping your controller make sure to check out Connection troubleshooting before you continue. If you have a connection with your controller it is time to setup your toolchain.

  1. In the Main Menu open File → Settings under Build, Execution, Deployment you will find the Toolchains menu.

  2. Add a new toolchain by pressing the + button and select Remote Host.

  1. In the Credentials press the ⚙️ button to enter the SSH configuration menu. Add a new configuration by pressing the + button and fill in the following data:
  • Host: 192.168.2.100
  • Port:22
  • User name: admin
  • Password: vectioneer

  1. Press the Test Connection button to make sure your connection is ok. If not re-check your controller connection.

  2. If the connection is good you can press OK to close the SSH Configurations menu.

  3. Wait for Make and the C/C++ Compiler to be detected then press the Apply button to save the Toolchain settings.

  4. Go CMake in the Settings menu and press the + button.

  1. Set the Build type to Debug and Toolchain to Remote Host.

  1. Press the OK button to save the CMake settings and close the settings window.

  2. Wait for +/- 20 seconds while a connection is being established, data is transferred and cmake is configured.

  3. Make sure that your controller is not running a Motorcortex application. Open the Terminal in Clion and run the the following command.

   ssh admin@192.168.2.100 'echo vectioneer | sudo -S motorcortex stop'
  1. In Clion set your Run/Debug Configuration to Debug-Remote Host.

  1. Press play to build and run the application.

  1. Congratulations you are now ready to start remote developing your Motorcortex application.

2.3 - Setup Cross Compilation

Set-up Cross Compilation (In Clion)

For cross compilation you use … (with remote device libraries).

Install Motorcortex toolchain on your system

For Cross compilation you will need a Motorcortex SDK you can download the SDK (Software Developement Kit) from the motorcortex.io store by pressing the download button below.

  1. Extract the archive and place the .sh file in a folder of your preference.

  2. In the terminal browse to the folder and run the .sh file from the terminal, for example:

   sh ./motorcortex-glibc-x86_64-motorcorcortex-image-dev-corei7-64-toolchain-X.X.X.sh
  1. You will be asked to enter the target directory where to install the SDK to. It is a good idea to keep the SDK in the home folder, where administrative rights are not required for example:
   ~/mcx-sdk

Mounting Sysroot on your system

The first step to Cross Compilation is to mount a Sysroot directory to your system that is used for cross-compilation. A `Sysroot is a directory which is considered to be the root directory for the purpose of locating headers and libraries of your application.

  1. Generate a new key pair with the following command.
   ssh-keygen -t rsa

  1. Use ssh-copy-id to provide a temporary access without a password to the remote Motorcortex host. If you are asked to fill in a password use vectioneer.
   ssh-copy-id admin@192.168.2.100

  1. Create a folder to mount the remote Motorcortex file system on with the following command.
   mkdir ~/mcx-sysroot
  1. Ensure that sshfs is installed with the following command:
   sudo apt-get install sshfs
  1. Mount the Motorcortex file system with the following command:
   sshfs admin@192.168.2.100:/ ~/mcx-sysroot/

Configure toolchain in clion.

Configure the compiler paths to use the Motorcortex cross-compilation toolchain.

  1. Open File → Settings → Build Execution Deployment → Toolchains.

  2. Create a new System Toolchain with the name motorcortex.

  3. Fill in Make, C Compiler and C++ Compiler paths to point the Motorcortex SDK as shown below.

  • Make:
   /home/ubuntu/mcx-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/make
  • C Compiler:
   /home/ubuntu/mcx-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc
  • C++ Compiler
   /home/ubuntu/mcx-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/x86_64-poky-linux/x86_64-poky-linux-g++

  1. Press Apply to save the changes.

Setup toolchain plugin

Configure sysroot path to use the headers and libraries from the remote host.

  1. Open File → Settings → Build, Execution, Deployment → **Motorcortex SDK and fill in the Sysroot path path the mounted folder (mcx-sysroot).
   home/ubuntu/mcx-sysroot

Add deployment connection to remote host (IS THIS STILL NEEDED IN THE NEW VERSION ALEXEY!!??)

  1. Add the connection to the remote Motorcortex host. Open Tools → Deployment → Configuration, add a new SFTP connection named mcx-host.

  2. select or add ssh configuration.

Press the Test Connection button to make sure that the filled data is correct.

Press the Autodetect button to set the root path. Uncheck the Visible only for this project checkbox make this host visible in all the projects.

press ok

Cross-compiling Motorcortex applications.

edit configurations.

select mcx dingie

save password checkbox

name cross-mcx-example

deployment profile to mxc-host

press ok

  1. In CLion open the project configuration menu.

  1. Click on + and select Motorcortex Remote Application. In the deployment profile field select Motorcortex remote host.

copy config folder

  1. In the Run/Debug configuration select the just created cross-compiled application with CrossDebug/CrossRelease build profile.

  2. Press the hammer button to cross-compile, play to copy and run on the remote Motorcortex host and the bug button to start a remote debug session.

Note if CMake CrossDebug/CrossRelease don’t work remove the “Motorcortex Remote Application” from Run/Debug Configurations and try again

2.4 - Install Integrated Development Environment

2.5 - Install Integrated Development Environment

2.6 - Setup Cross Compilation

2.7 - Setup Cross Compilation

2.8 - Setup Remote Development

2.9 - Setup Remote Development

3 - Motorcortex Application Template

This section will explain you how to create a new poject and how the Motorcortex project template works.

In Clion there is a template avaiable to create a new C++ Motorcortex project. This template provides you a structure to get started creating a control application and will consist of the following ellements:

  • The config folder includes saved values of the module’s parameters and EtherCAT configuration.
  • The control folder contains a template of the main control loop Module. This where all the control computation happens.
  • The logic folder (optional) contains a template of the Finite State Machine to build advanced logic.
  • A main.cpp file, where all system and task configuration and linking of parameters happens. The generated main.cpp already has all the required code, necessary to run a Hard-Real Time application and communication server.

This section will explain how to create a new C++ Motorcortex project, and explain the elements provided by the template.

Creating a new C++ Motorxortex Project

  1. In the CLion main menu select File → New → C++ Motorcortex. Then enter a new location for you project and projectname. If your project requires a logic Tasks, check the box to generate a Logic template.

The generated template provided the following structure:

  • The config folder
  • The control folder
  • The logic folder (optional)
  • A main.cpp file

When developing remotely you will have to set your toolchain to remote host when creating a new project. When you use crosscomplilation ignore steps 2 and 3.

  1. In the Main Menu open go to File → Settings.

  1. go to CMake toolchain select remote host.

The Config Folder

Configuring your C++ Motorcortex project has to be done in the config folder. Here you can configurate EtherCAT-devices, Controller Configuration in the config.json file and manage your certificate.

control

In the control folder you will find the control.xml in this xml file you can.

io

In the io folder you can configurate your io devices. this has to be done configuring the master.xml and topology.xml

master.xml

topology.xml

config.json

demo-cert.pem

The Control Folder

The Logic Folder

Main.cpp

The main.cpp is

Programming Concepts

Modules and Tasks

A Motorcortex application is built around the concept of Modules and Tasks. A Module is an object that performs calculations based on inputs and internal variables and produces outputs. Inputs are either set directly from a higher level Module (via a setter function) or are set via the Parameter Tree.

Below a template for a Module header file is shown.

//Module header
#ifndef MAINCONTROLLOOP_H
#define MAINCONTROLLOOP_H
#include <mcx/mcx_core.h>

class MainControlLoop : public mcx::container::Module {
public:
  MainControlLoop() = default;
  ~MainControlLoop() override = default;
private:
  void create_(const char* name, mcx::parameter_server::Parameter* parameter_server, uint64_t dt_micro_s) override;
  bool initPhase1_() override;
  bool initPhase2_() override;
  bool startOp_() override;
  bool stopOp_() override;
  bool iterateOp_(const mcx::container::TaskTime& system_time, mcx::container::UserTime* user_time) override;
  double myDouble_{0};
};

#endif /* MAINCONTROLLOOP_H */

The module is a passive object, which means that it does not have an execution thread and must be executed by a Task object. A Task plays the role of an execution thread. Modules can be registered in the Task, which will allocate required resources, like CPU, scheduler policy, priority and start to execute modules sequentially in a loop.

The Module State Machine

All Motorcortex modules have a certain structure, which describes the life-cycle of the module. Modules go through different phases of the life-cycle during startup according to the schematic below.

   Module:                                        Task:
   ┌──────────────────────────────┐                        
   │   Not Initialized            │                 
   └───┬──────────────────────────┘               ┌─────────────────────────────────────┐    
       │ (Event: create)                          │ Creates modules and submodules.     │
   ┌───v───────┐                                  └───┬─────────────────────────────────┘
   │   Phase0  │                                      │ (Event: configure)
   └───┬───────┘                                  ┌───v─────────────────────────────────┐
       │ (Event: initPhase1)                      │ Add parameters to the tree          │  
       │                                          │                                     │           
   ┌───v───────┐                                  └───┬─────────────────────────────────┘
   │   Phase1  │                                      │ (Event: start)  
   └───┬───────┘                                  ┌───v─────────────────────────────────┐  
       │ (Event: initPhase2)                      │ Parameter tree is ready,            │
       │                                          │                                     │
   ┌───v──────────────────────────┐               └───┬─────────────────────────────────┘           
   │   Phase2                     │                   │    
   └───┬──────────────────────^───┘               ┌───v─────────────────────────────────┐
       │ (Event: startOp)     │ (Event: stopOp)   │  Real-time event loop is            │ 
       │                      │                   │  ready to start.                    │
   ┌───v──────────────────────┴───┐               └─────────────────────────────────────┘
   │   Operation                  │                  
   └───┬──────────────────────────┘  
       │ (Event: exit)
   ┌───v───────┐
   │ Destroyed │
   └───────────┘

Modules are first created by calling the Module’s create function. This in turn calls the create method of all sub-modules to register all the Modules in the Parameter Tree.

Then the Modules are added to a Task and the Task’s configure method is called. This calls initPhase1 of all the (sub-)modules. In initPhase1 all (sub-)modules shall add their Parameters to the Parameter Tree.

Now that the Tree is complete and running (Phase1), Parameter values can be loaded from a file.

Then the Task can be started by calling it’s start method. This calls the initPhase2 methods of all (sub-)modules and then startOp, just before the task switches to realtime mode.

In operation state, the system cyclically calls the Module’s iterateOp method that calculates the new state in each timestep. In the iterateOp method all submodules also need to be iterated.

Creating submodules (create)

Submodules are created in the create method of their parent Module, like shown below. The submodule then registers its own parameters and submodules in the tree.

void MainControlLoop::create_(const char* name,
  mcx::parameter_server::Parameter* parameter_server,
  uint64_t dt_micro_s) {
  createSubmodule(&mySubmodule, "mySubmodule");
}

Registering Parameters (initPhase1)

In initPhase1 the Module’s parameters need to be registered into the Parameter Tree.

bool MainControlLoop::initPhase1_() {
  using ParameterType = mcx::parameter_server::ParameterType;
  addParameter("input", ParameterType::INPUT,&input_);
  addParameter("gain", ParameterType::PARAMETER, &gain_);
  addParameter("output", ParameterType::OUTPUT, &output);
  return true;
}

The ParameterType can be set to one of the types as specified in paragraph Parameter Types in Motorcortex.

Optional callbacks for special cases (initPhase2, startOp, stopOp)

The initPhase2 method is in general not used. In special cases, it can be used for additional initializations, memory allocations, validations of the loaded parameters.

The startOp and stopOp methods can be used to set and reset module state during start/stop routine.

It is not recommended to put any delays or blocking calls in these callbacks, because they can cause a significant delay during module startup.

Iterating (iterateOp)

Calculations shall be done in the Module’s iterateOp method. The iterateOp method of the top-level Module is called cyclically by the corresponding Task. The top-level module then iterates submodules from its own iterateOp method. For instance:

bool MainControlLoop::iterateOp_(const container::TaskTime& system_time, container::UserTime* user_time) {
   submodule.setInput(input);
   submodule.iterate(system_time, user_time);
   double subOutput = submodule.getOutput();
   output_ = gain_ * input_ + subOutput * getDtSec();
   return true;
}

Usually, for a submodule, first the inputs are set, then the module is iterated, then the outputs are read.

For a lot of digital control systems the step size is required in calculations (e.g. for calculating filters or integrators). The application cycle time in seconds can be obtained by calling the getDtSec() function. The getDtSec function returns the configured cycle time of the Task, not the actual cycle time (which may vary a small amount each cycle).

Overriding inputs and outputs

If the Module’s inputs are set via the Parameter Tree, the inputs can be overridden (forced) at runtime. Outputs can also be overridden before they are updated into the Parameter Tree. The output overrides have no effect on the actual variable value of the module. Also a parameter tree input might have no effect if its associated variable is set directly via a setter function.

Application Configuration.

Tasks

Tasks provide the execution environment of Motorcortex Modules. Each Task can manage multiple Modules. A Task also switches the Modules through the Module State Machine.

The iterate methods of all the Modules inside a Task are called cyclically with the update rate that was specified when the Task is created. The Modules are iterated sequentially in the order they were added to the Task.

 // creates a Module
 mcx::log::Module myModule();
 myModule.create("myModule", &param_server, rt_dt_micro_s);
 // create and configure the task
 mcx::container::Task myTask("myTask", &param_server);
 myTask.add(&myModule);
 myTask.configure();
 ...
 myTask.start(rt_dt_micro_s, container::TaskSched::REALTIME, {0}, 80);

The Logger Task

The Logger is a standard Module that provides logging functions for other Modules to write messages to certain log-levels. The standard log levels are:

  • LOG_DEBUG
  • LOG_INFO
  • LOG_WARNING
  • LOG_ERROR

The log is accessible from inside any module by calling the respective log function. E.g. LOG_INFO(“%d red balloons”,99). The LOG_* functions provide similar syntax to the standard C function printf. An example how the logger Task is created is shown below:

 // creates log output to a file
 mcx::log::Module logger(path_log);
 logger.create("logger", &param_server, rt_dt_micro_s);
 // create and configure log output task
 mcx::container::Task logger_task("Logger_task", &param_server);
 logger_task.add(&logger);
 logger_task.configure();

4 - Realtime Configuration

Introduction

MOTORCORTEX is designed for running Tasks with hard-realtime deadlines. It does this without using special hypervisors that execute code within their own environment and scheduler. MOTORCORTEX uses the realtime capabilities of the latest standard Linux kernels, which makes it possible to run userspace applications with very stringent hard-realtime deadlines. For optimal performance, Vectioneer provides specially tuned Linux distributions that have been tested on a wide range of hardware, like the CX range of industrial embedded computers from Beckhoff, Compulab’s Fitlet2, the Aaeon UP², Siemens Simatic IPC127E IOT gateways and even System-On-Chips (SoC) like the Raspberry Pi or NanoPi.

Since MOTORCORTEX applications are user-space applications they can also be compiled to run in non-realtime on any platform that supports Linux. This may be sufficient for a lot of applications that only need soft-realtime.

In this chapter it will be shown how to configure a MOTORCORTEX application for hard-realtime and give some pointers how to get the best possible performance and what to avoid.

The do’s and don’ts of hard-realtime

Hard-realtime is very difficult to achieve in a general purpose operating systems, especially on modern CPUs or systems that have a lot of features that may interrupt the realtime process. Especially hardware interrupts always have been a problem for realtime, like interacting with USB devices or video cards. Modern systems are generally tuned for speed (or responsiveness) and not for realtime (deterministic) behavior; snappy response to user inputs for instance is then prioritized over calculations done in the background.

Luckily, in Linux this realtime or low-latency requirement has been taken very seriously by Kernel developers and a lot of development has already gone into making Linux a general purpose operating system with uncompromised real-time capability. This originated with the PREEMPT_RT patch that was originally published by Thomas Gleixner and Ingo Molnar, which currently has reached maturity to be included into the standard (Vanilla) Linux Kernel.

Currently, with a tuned Linux kernel the same performance can be achieved as with dedicated Realtime Operating Systems or hypervisors.

In general, to make a realtime Task run such that it never misses a beat, follow the following rules:

  • (De-)Allocate memory before realtime is needed. Reduce or better eliminate dynamic memory allocations during an iterate task (e.g. avoid using new in C++). Do not use blocking calls; for example for IO calls. Be careful with using external libraries in your code that are not designed to be real-time-capable.
  • Reduce the amount of page-faults as much as possible.
  • Do not use the CPU’s C-States. C-States are there to put system into (partial) sleep or reduce clock speeds when the system load allows this. When this happens probably realtime performance will be badly affected. Disable C-States in the BIOS. The negative consequence of this is that the system will consume more power and generate more heat than usual. Check if the system is sufficiently cooled.
  • Some CPUs throttle their throughput when the temperature rises above some point. Make sure there is adequate cooling for the task, and test realtime performance under full load in a worst-case environment.

The good news is that MOTORCORTEX already takes care of a lot of tuning and configuration for you.

Assigning Tasks to CPUs

To run tasks in realtime, the CPUs that are going to be used for realtime must be isolated, so other processes cannot use these CPUs.

// start low latency, isolate CPU 0 and 1
utils::startRealTime({0, 1});

As soon as these CPU cores are isolated, other tasks are pushed off these CPUs onto other CPUs. Make sure that not all CPUs are isolated, Linux itself needs at least one CPU to run.

Tasks can be configured to run with different schedulers. Currently the following scheduler policies are supported:

enum class TaskSched {
 NORMAL = SCHED_OTHER, // Default non-realtime policy.
 REALTIME = SCHED_FIFO, // Default realtime policy.
 REALTIME_FIFO = SCHED_FIFO, // Realtime FIFO policy.
 REALTIME_RR = SCHED_RR // Realtime Round-Robin policy.
};

When a task is started it can be assigned to a scheduler, to a number of CPUs, with a certain priority. In general, for non-realtime tasks the NORMAL scheduler is used and for realtime tasks the REALTIME scheduler is used:

// start logger and communication with non-realtime scheduler
logger_task.start(rt_dt_micro_s,
container::TaskSched::NORMAL);
comm_task.start(rt_dt_micro_s, container::TaskSched::NORMAL);
// start control and ethercat with realtime scheduler,
// attach to isolated CPUs 0 and 1, set priorities to 80
controls_task.start(rt_dt_micro_s,
container::TaskSched::REALTIME, {0}, 80);
ethercat_task.start(rt_dt_micro_s,
container::TaskSched::REALTIME, {1}, 80);

Adjusting timing with secaligned

Each task has a secaligned parameter that can be modified through the MOTORCORTEX Parameter Tree. The secalign parameter aligns the Task’s execution cycle with respect to the system monotonic timer. The secaligned value can be set in the normalized range of [0..1], where 1 corresponds to a full cycle time of the task.

In general, changing secaligned from its default value is not required.

TBD: Diagram & further explanation

Parallelizing execution; splitting the load into Workers

MOTORCORTEX has a special active container to run callable targets asynchronously. This can be used to parallelize heavy computations inside the control blocks or perform blocking calls asynchronously (without blocking the control loop). The advantage of the MOTORCORTEX Worker is that it can be setup to run on a specific CPU with specific priority and scheduler, by default the Worker inherits these properties from the calling task. It is good practice to create the worker in the startOp phase, when the calling task has all its Realtime related properties set.

bool MainControlLoop::startOp_() {
  // creates the Worker with the same priority and CPU affinity as the calling task
  worker1.create(std::string("worker1");
  // creates the Worker with the Normal priority on the non-isolated CPUs
  worker2.create("worker2", mcx::container::TaskSched::NORMAL, mcx::utils::getAvailableCpu());
  return true;
}
bool MainControlLoop::iterateOp_(const container::TaskTime& system_time, container::UserTime* user_time) {
  // moving half of the control loops to the worker
  unsigned int split_the_work = number_of_control_loops / 2;
  for (unsigned int cnt = 0; cnt < split_the_work; cnt++) {
    worker1.start([&, cnt]() {
      punchControl_[cnt].iterate(system_time, user_time);
    });
  }
  // computing the other half in the task
  for (unsigned int cnt = split_the_work; cnt < number_of_control_loops; cnt++) {
    punchControl_[cnt].iterate(system_time, user_time);
  }
  // Important: wait for the worker to finish
  worker1.join(0);
}

5 - Debugging

Cycletime and utilization

Cycle Time and Utilization

MOTORCORTEX Tasks show statistical information in the Parameter Tree that can indicate problems with performance:

  • absolute_cycle_max; maximum cycle time in microseconds since the last reset
  • actual_cycle_max; maximum cycle time over the last second
  • desired_cycle_time_micro_s; the target cycle time in microseconds
  • reset_absolute_cycle_max; if set to 1 the absolute_cycle_max is reset to zero
  • utilization_max; the amount of time the task takes in percentage of the desired_cycle_time_micro_s

Linux debugging tools

There are a number of Linux tools that can deep-dive into your application and the system:

6 - Motorcortex APIs

Motorcortex provides a number of libraries with different features:

  • mcx-core;
    the mcx-core library is required as a minimum for a Motorcortex application, it provides a convenient abstraction for Hard-Realtime applications, EtherCAT and secure Communication Middleware.

  • mcx-math;
    mcx-math provides common math objects suitable for use in realtime applications, like a matrix class, rotation, 6DOF pose and helper functions to convert between reference frames

  • mcx-control;
    mcx-control provides object for control systems, such as filters, limiters, switches, PID controller, faders, signal generators

  • mcx-mechanics;
    the mcx-mechanics library provides mechanical function blocks, like serial and parallel forward and inverse kinematics and dynamics, rigid body simulation

  • mcx-electrics;
    mcx-electrics provides currently simulation blocks to simulate an encoder, an electric motor and a emergency stop relay.

  • mcx-hydraulics;
    mcx-hydraulics provides function blocks for hydraulic systems, like accurate simulation of hydraulic valves and actuators and control blocks like a hydraulic control loop, valve correction

  • mcx-motion;
    mcx-motion provides a path planner and a motion interpreter used for robot control

7 -

Developing Motorcortex Applications in CPP

Introduction

MOTORCORTEX provides out of the box solution to develop Hard-Realtime Control Applications in C++ with an integrated high-speed and secured communication for distributed systems. It supports popular industrial buses, like EtherCAT.

The Parameter Tree

The MOTORCORTEX Parameter Tree

In a MOTORCORTEX application all control Modules are organized in a tree structure. The Parameter Tree contains a snapshot of all the inputs, outputs and internal data of the control objects at the current time.

Each control Module is represented as a subtree (folder). Modules can be nested. In a general control application there is usually a “Logic”, “Control” and also an “EtherCAT” folder that represents the data of the corresponding tasks.

Inside the folders the data of the associated Module is contained. MOTORCORTEX currently supports the following datatypes in the Tree: boolean, integer, double or binary (e.g. a C-struct). Also arrays of a single datatype are supported. Array elements start at index 0 (zero).

The MOTORCORTEX-Desk web-application can be used to connect to a MOTORCORTEX application

Parameter Types in MOTORCORTEX

This subsection shortly describes how the different parameter types can be used in combination with EtherCAT.

The MOTORCORTEX structure has the following parameter types:

  • Input: In this parameter type a value can be written. In MOTORCORTEX-Desk an input is depicted as .
  • Output: this parameter type is read-only. In MOTORCORTEX-Desk an output is depicted as .
  • Parameter: This is parameter type that is used to configure the system. Typical uses are parameters and system constants. Parameters are retrieved from the parameter list during start-up. If the user changes a value and wants to use in the future, he has to save the list before rebooting/switching off the controller. A parameter can be read and written. In MOTORCORTEX-Desk an parameter is depicted as .
  • Persistent: these are values that contain process data that is continuously updated and will be retained through application restarts. Typical applications are running ours or amount of produced products. In MOTORCORTEX-Desk an persistent is depicted as .

Setting up a development environment

Introduction

Vectioneer uses CMake (https://cmake.org/) as its main tool for building, testing and deploying software. Also MOTORCORTEX itself is built using CMake. CMake is widely accepted and supported by many IDEs, which means that the user is free to choose his/her own development environment.

To make things more convenient, Vectioneer has developed plugins for CLion (https://www.jetbrains.com/clion). CLion is an advanced Integrated Development Environment that has many tools that make programming easy, is available across platforms and is capable to cross-compile and deploy MOTORCORTEX applications straight to the controller with a click of a button.

This chapter is a guide, how to install CLion and the MOTORCORTEX plugins from the CLion Marketplace and get started quickly with developing MOTORCOREX applications.

Installing CLion

Download CLion for your platform from: https://www.jetbrains.com/clion/download;

Run the install script.

Installing the MOTORCORTEX Templates plugin

Start CLion.

In the Main Menu open: File → Settings, then find the Plugins menu.

CLion plugins menu

In the search section type motorcortex, then click Install.

Motorcortex Templates Plugin

The MOTORCORTEX development toolchain

There are two options how to develop MOTORCORTEX applications:

  • use remote development and compile the application on the MOTORCORTEX Target System;
  • cross-compile on your development PC and deploy the executable to the MOTORCORTEX Target System

The first method is easy to set up, it does not require additional dependencies however the downside is that a compile time for the large projects may be significant. The second method overcomes this problem, but requires a cross-compilation tool-chain to be set up.

Setting up remote development in CLion

In the Main Menu open: File → Settings, find the Toolchains menu.

CLion Toolchains menu

Select Environment type: Remote Host. Enter the remote host credentials and press OK, wait for a few seconds to let CLion find all the necessary tools.

Configure Remote Credentials

The remote toolchain is now ready to use.

Setting up cross-compilation in CLion (Linux)

Download the development toolchain from our public git repository:

https://git.vectioneer.com/pub/motorcortex-sdk

To download latest snapshot, click on the button marked with red colour.

Extract downloaded archive to the local folder. Open the Terminal and execute:

motorcortex-glibc-x86_64-motorcorcortex-image-dev-corei7-64-toolchain-X.X.X.sh

Enter the target directory where to install the SDK to. It is a good idea to keep the SDK in the home folder, where administrative rights are not required: (for example ~/mcx-sdk):

Start CLion and install the MOTORCORTEX Toolchain plugin. To install the plugin, open File → Settings → Plugins. Then, in the Marketplace section, type “motorcortex”. Install the plugin and restart CLion.

Installing the MOTORCORTEX Toolchain plugin

Before working with a remote MOTORCORTEX host, make sure that the host is connected and reachable.

Use ssh-copy-id to provide a temporary access without a password to the remote MOTORCORTEX host.

ssh-copy-id <admin@192.168.2.100>

If there is an error: No identities found, run ssh-keygen to **generate a new key pair and run ssh-copy-id command again.

ssh-keygen -t rsa

Open Terminal and create a folder to mount the remote MOTORCORTEX file system on.

mkdir \~/mcx-sysroot

Mount MOTORCORTEX file system with the following command:

sshfs admin@192.168.2.100:/ \~/mcx-sysroot/

Configure the compiler paths to use the MOTORCORTEX cross-compilation toolchain.

Open File → Settings → Build, Execution, Deployment → Toolchains. Create a new System Toolchain with the name motorcortex.

Fill in Make, C Compiler and C++ Compiler paths to point the MOTORCORTEX SDK as shown below.

Configure sysroot path to use the latest MOTORCORTEX libraries from the remote host.

Open File → Settings → Build, Execution, Deployment → Motorcortex SDK and fill in the path to the mounted folder (mcx-sysroot) in the field “Sysroot path”. The other field on this screen are left empty.

Add the connection to the remote MOTORCORTEX host. Open Tools → Deployment → Configuration menu, add a new SFTP connection and fill in the details (see below). Press the Test Connection button to make sure that the filled data is correct. Press the Autodetect button to set the root path. Uncheck the Visible only for this project checkbox make this host visible in all the projects.

Cross-compiling MOTORCORTEX applications.

In CLion open the project configuration menu.

Click on + and select Motorcortex Remote Application. In the deployment profile field select MOTORCORTEX remote host.

In the Run/Debug configuration select the just created cross-compiled application with CrossRelease or CrossDebug build profile. Press the hammer button to cross-compile, play to copy and run on the remote MOTORCORTEX host and the bug button to start a remote debug session.

Deployment and Auto-start of applications.

TBD

Alternative build environments and IDEs

Since the MOTORCORTEX development environment uses CMake as its main tool, development can also easily be setup in other IDEs like Eclipse, NetBeans or Microsoft Visual Code. Also CMake can be used from the command line if one wishes. Refer to your IDE’s documentation on how to setup your environment to work with CMake.

Code can always be compiled on the Vectioneer Linux Target System, since that has everything on-board to compile MOTORCORTEX projects. In general you can build the application by using the default vuild method of clion:

Copy your project onto the Target System (e.g. /home/admin/myProject/).

Then login to the Target system via SSH (on a Windows system you can use Putty).

Navigate to the Project folder:

cd /home/admin/myProject

Make an empty build folder and enter it:

mkdir build
cd build

Run cmake, compile and install:

cmake ..
make
sudo make install

Generating a template for a MOTORCORTEX Application

It easy to create a new MOTORCORTEX application from the existing template.

In the CLion main menu select File → New → C++ Motorcortex. Then enter a new location for you project. If your project requires a logic Task, check the box to generate a Logic template.

Generating a Motorcortex application from a template

The generated template has the following structure:

  • A main.cpp file, where all system and task configuration and linking of parameters happens. The generated main.cpp already has all the required code, necessary to run a Hard-Real Time application and communication server.
  • the control folder contains a template of the main control loop Module. This where all the control computation happens.
  • the logic folder (optional) contains a template of the Finite State Machine to build advanced logic.
  • the config folder includes saved values of the module’s parameters and EtherCAT configuration.

Source tree structure

Programming Concepts

Modules and Tasks

A MOTORCORTEX application is built around the concept of Modules and Tasks. A Module is an object that performs calculations based on inputs and internal variables and produces outputs. Inputs are either set directly from a higher level Module (via a setter function) or are set via the Parameter Tree.

Below a template for a Module header file is shown.

//Module header
#ifndef MAINCONTROLLOOP_H
#define MAINCONTROLLOOP_H
#include <mcx/mcx_core.h>

class MainControlLoop : public mcx::container::Module {
public:
  MainControlLoop() = default;
  ~MainControlLoop() override = default;
private:
  void create_(const char* name, mcx::parameter_server::Parameter* parameter_server, uint64_t dt_micro_s) override;
  bool initPhase1_() override;
  bool initPhase2_() override;
  bool startOp_() override;
  bool stopOp_() override;
  bool iterateOp_(const mcx::container::TaskTime& system_time, mcx::container::UserTime* user_time) override;
  double myDouble_{0};
};

#endif /* MAINCONTROLLOOP_H */

The module is a passive object, which means that it does not have an execution thread and must be executed by a Task object. A Task plays the role of an execution thread. Modules can be registered in the Task, which will allocate required resources, like CPU, scheduler policy, priority and start to execute modules sequentially in a loop.

The Module State Machine

All MOTORCORTEX modules have a certain structure, which describes the life-cycle of the module. Modules go through different phases of the life-cycle during startup according to the schematic below.

   Module:                                        Task:
   ┌──────────────────────────────┐                        
   │   Not Initialized            │                 
   └───┬──────────────────────────┘               ┌─────────────────────────────────────┐    
       │ (Event: create)                          │ Creates modules and submodules.     │
   ┌───v───────┐                                  └───┬─────────────────────────────────┘
   │   Phase0  │                                      │ (Event: configure)
   └───┬───────┘                                  ┌───v─────────────────────────────────┐
       │ (Event: initPhase1)                      │ Add parameters to the tree          │  
       │                                          │                                     │           
   ┌───v───────┐                                  └───┬─────────────────────────────────┘
   │   Phase1  │                                      │ (Event: start)  
   └───┬───────┘                                  ┌───v─────────────────────────────────┐  
       │ (Event: initPhase2)                      │ Parameter tree is ready,            │
       │                                          │                                     │
   ┌───v──────────────────────────┐               └───┬─────────────────────────────────┘           
   │   Phase2                     │                   │    
   └───┬──────────────────────^───┘               ┌───v─────────────────────────────────┐
       │ (Event: startOp)     │ (Event: stopOp)   │  Real-time event loop is            │ 
       │                      │                   │  ready to start.                    │
   ┌───v──────────────────────┴───┐               └─────────────────────────────────────┘
   │   Operation                  │                  
   └───┬──────────────────────────┘  
       │ (Event: exit)
   ┌───v───────┐
   │ Destroyed │
   └───────────┘

Modules are first created by calling the Module’s create function. This in turn calls the create method of all sub-modules to register all the Modules in the Parameter Tree.

Then the Modules are added to a Task and the Task’s configure method is called. This calls initPhase1 of all the (sub-)modules. In initPhase1 all (sub-)modules shall add their Parameters to the Parameter Tree.

Now that the Tree is complete and running (Phase1), Parameter values can be loaded from a file.

Then the Task can be started by calling it’s start method. This calls the initPhase2 methods of all (sub-)modules and then startOp, just before the task switches to realtime mode.

In operation state, the system cyclically calls the Module’s iterateOp method that calculates the new state in each timestep. In the iterateOp method all submodules also need to be iterated.

Creating submodules (create)

Submodules are created in the create method of their parent Module, like shown below. The submodule then registers its own parameters and submodules in the tree.

void MainControlLoop::create_(const char* name,
  mcx::parameter_server::Parameter* parameter_server,
  uint64_t dt_micro_s) {
  createSubmodule(&mySubmodule, "mySubmodule");
}

Registering Parameters (initPhase1)

In initPhase1 the Module’s parameters need to be registered into the Parameter Tree.

bool MainControlLoop::initPhase1_() {
  using ParameterType = mcx::parameter_server::ParameterType;
  addParameter("input", ParameterType::INPUT,&input_);
  addParameter("gain", ParameterType::PARAMETER, &gain_);
  addParameter("output", ParameterType::OUTPUT, &output);
  return true;
}

The ParameterType can be set to one of the types as specified in paragraph Parameter Types in MOTORCORTEX.

Optional callbacks for special cases (initPhase2, startOp, stopOp)

The initPhase2 method is in general not used. In special cases, it can be used for additional initializations, memory allocations, validations of the loaded parameters.

The startOp and stopOp methods can be used to set and reset module state during start/stop routine.

It is not recommended to put any delays or blocking calls in these callbacks, because they can cause a significant delay during module startup.

Iterating (iterateOp)

Calculations shall be done in the Module’s iterateOp method. The iterateOp method of the top-level Module is called cyclically by the corresponding Task. The top-level module then iterates submodules from its own iterateOp method. For instance:

bool MainControlLoop::iterateOp_(const container::TaskTime& system_time, container::UserTime* user_time) {
   submodule.setInput(input);
   submodule.iterate(system_time, user_time);
   double subOutput = submodule.getOutput();
   output_ = gain_ * input_ + subOutput * getDtSec();
   return true;
}

Usually, for a submodule, first the inputs are set, then the module is iterated, then the outputs are read.

For a lot of digital control systems the step size is required in calculations (e.g. for calculating filters or integrators). The application cycle time in seconds can be obtained by calling the getDtSec() function. The getDtSec function returns the configured cycle time of the Task, not the actual cycle time (which may vary a small amount each cycle).

Overriding inputs and outputs

If the Module’s inputs are set via the Parameter Tree, the inputs can be overridden (forced) at runtime. Outputs can also be overridden before they are updated into the Parameter Tree. The output overrides have no effect on the actual variable value of the module. Also a parameter tree input might have no effect if its associated variable is set directly via a setter function. Application Configuration

Tasks

Tasks provide the execution environment of MOTORCORTEX Modules. Each Task can manage multiple Modules. A Task also switches the Modules through the Module State Machine.

The iterate methods of all the Modules inside a Task are called cyclically with the update rate that was specified when the Task is created. The Modules are iterated sequentially in the order they were added to the Task.

 // creates a Module
 mcx::log::Module myModule();
 myModule.create("myModule", &param_server, rt_dt_micro_s);
 // create and configure the task
 mcx::container::Task myTask("myTask", &param_server);
 myTask.add(&myModule);
 myTask.configure();
 ...
 myTask.start(rt_dt_micro_s, container::TaskSched::REALTIME, {0}, 80);

The Parameter Tree

In a MOTORCORTEX application all Modules and variables are organized in a tree structure; the Parameter Tree. The Parameter Tree is used to communicate data between tasks and to the outside world in a thread-safe manner. In the tree the modules are represented as subtrees (folder) and variables are represented as leafs.

The Parameter Tree contains a snapshot of all the registered inputs, outputs and internal data of the modules at the current time. Each Task or Module can create its own subtrees (folder) in the Parameter Tree. Modules can be nested and register their own parameters in the Tree.

The Parameter Tree always starts from the root node, which is passed to the modules and tasks that then can register their own parameters or create submodules that register their own data.

  parameter_server::Parameter param_server;
  // creates root of the parameter tree
  param_server.create("root", nullptr);
  // creates log output to a file
  mcx::log::Module logger(path_log);
  logger.create("logger", &param_server, rt_dt_micro_s);
  // create and configure log output task
  mcx::container::Task logger_task("Logger_task", &param_server);
  logger_task.add(&logger);
  logger_task.configure();

The Communication Server

The Communication Server provided by MOTORCORTEX has two parts:

  1. the first part is a Signaling Server which uses a Request/Reply reliable messaging pattern.
  2. The second part is a Publishing Server or Publisher which uses a Publish/Subscribe best-effort delivery messaging pattern.

A Signaling Server is required for Remote Procedure Calls and for managing the control application, for example updating parameter values. It is also responsible for managing the Publishing Server.

The Publishing Server can organize data into groups and publish them with the requested frequency. The Publisher is used to send Real-time data continuously.

The Signaling Server is set up in the main(…) function and runs until a terminating signal is received.

  // creates req/rep server
  comm::RequestReply reqrep;
  // when all modules are configured req/rep caches the parameter tree
  reqrep.configure(&param_server, path_control_dir);
  bool is_connected = reqrep.start(cmd_args.conn_data);
  ASSERT(is_connected, "Failed to start Req/Rep server");
  // running until terminate signal is received
  while (utils::running()) {
    reqrep.iterate();
  }

Because the Publisher has a cyclic behavior, it is added to a Task that executes it with the given update rate. An example how the publisher is created is shown below:

 // creates the req/rep server
 comm::RequestReply reqrep;
 // creates the publisher module
 comm::Publisher publisher(reqrep, cmd_args.conn_data);
 publisher.create("ParamPub", &param_server, rt_dt_micro_s);
 // creates and configures the communication task
 container::Task comm_task("Comm_task", &param_server);
 comm_task.add(&publisher);
 comm_task.configure();
 // when all modules are configured req/rep caches the parameter tree
 reqrep.configure(&param_server, path_control_dir);
 comm_task.start(rt_dt_micro_s, container::TaskSched::NORMAL);
 ... // start other tasks
 bool is_connected = reqrep.start(cmd_args.conn_data);
 ASSERT(is_connected, "Failed to start Req/Rep server");
 // req/rep server is used to create the main loop because it is blocking
 while **(utils::running()) {
   reqrep.iterate();
 }
 // stop req/reply server
 reqrep.stop();
 // stop all tasks
 comm_task.stop();
 ...

The Logger Task

The Logger is a standard Module that provides logging functions for other Modules to write messages to certain log-levels. The standard log levels are:

  • LOG_DEBUG
  • LOG_INFO
  • LOG_WARNING
  • LOG_ERROR

The log is accessible from inside any module by calling the respective log function. E.g. LOG_INFO(“%d red balloons”,99). The LOG_* functions provide similar syntax to the standard C function printf.

An example how the logger Task is created is shown below:

 // creates log output to a file
 mcx::log::Module logger(path_log);
 logger.create("logger", &param_server, rt_dt_micro_s);
 // create and configure log output task
 mcx::container::Task logger_task("Logger_task", &param_server);
 logger_task.add(&logger);
 logger_task.configure();

Configuring Realtime and Performance

Introduction

MOTORCORTEX is designed for running Tasks with hard-realtime deadlines. It does this without using special hypervisors that execute code within their own environment and scheduler. MOTORCORTEX uses the realtime capabilities of the latest standard Linux kernels, which makes it possible to run userspace applications with very stringent hard-realtime deadlines. For optimal performance, Vectioneer provides specially tuned Linux distributions that have been tested on a wide range of hardware, like the CX range of industrial embedded computers from Beckhoff, Compulab’s Fitlet2, the Aaeon UP², Siemens Simatic IPC127E IOT gateways and even System-On-Chips (SoC) like the Raspberry Pi or NanoPi.

Since MOTORCORTEX applications are user-space applications they can also be compiled to run in non-realtime on any platform that supports Linux. This may be sufficient for a lot of applications that only need soft-realtime.

In this chapter it will be shown how to configure a MOTORCORTEX application for hard-realtime and give some pointers how to get the best possible performance and what to avoid.

The do’s and don’ts of hard-realtime

Hard-realtime is very difficult to achieve in a general purpose operating systems, especially on modern CPUs or systems that have a lot of features that may interrupt the realtime process. Especially hardware interrupts always have been a problem for realtime, like interacting with USB devices or video cards. Modern systems are generally tuned for speed (or responsiveness) and not for realtime (deterministic) behavior; snappy response to user inputs for instance is then prioritized over calculations done in the background.

Luckily, in Linux this realtime or low-latency requirement has been taken very seriously by Kernel developers and a lot of development has already gone into making Linux a general purpose operating system with uncompromised real-time capability. This originated with the PREEMPT_RT patch that was originally published by Thomas Gleixner and Ingo Molnar, which currently has reached maturity to be included into the standard (Vanilla) Linux Kernel.

Currently, with a tuned Linux kernel the same performance can be achieved as with dedicated Realtime Operating Systems or hypervisors.

In general, to make a realtime Task run such that it never misses a beat, follow the following rules:

  • (De-)Allocate memory before realtime is needed. Reduce or better eliminate dynamic memory allocations during an iterate task (e.g. avoid using new in C++). Do not use blocking calls; for example for IO calls. Be careful with using external libraries in your code that are not designed to be real-time-capable.
  • Reduce the amount of page-faults as much as possible.
  • Do not use the CPU’s C-States. C-States are there to put system into (partial) sleep or reduce clock speeds when the system load allows this. When this happens probably realtime performance will be badly affected. Disable C-States in the BIOS. The negative consequence of this is that the system will consume more power and generate more heat than usual. Check if the system is sufficiently cooled.
  • Some CPUs throttle their throughput when the temperature rises above some point. Make sure there is adequate cooling for the task, and test realtime performance under full load in a worst-case environment.

The good news is that MOTORCORTEX already takes care of a lot of tuning and configuration for you.

Assigning Tasks to CPUs

To run tasks in realtime, the CPUs that are going to be used for realtime must be isolated, so other processes cannot use these CPUs.

// start low latency, isolate CPU 0 and 1
utils::startRealTime({0, 1});

As soon as these CPU cores are isolated, other tasks are pushed off these CPUs onto other CPUs. Make sure that not all CPUs are isolated, Linux itself needs at least one CPU to run.

Tasks can be configured to run with different schedulers. Currently the following scheduler policies are supported:

enum class TaskSched {
 NORMAL = SCHED_OTHER, // Default non-realtime policy.
 REALTIME = SCHED_FIFO, // Default realtime policy.
 REALTIME_FIFO = SCHED_FIFO, // Realtime FIFO policy.
 REALTIME_RR = SCHED_RR // Realtime Round-Robin policy.
};

When a task is started it can be assigned to a scheduler, to a number of CPUs, with a certain priority. In general, for non-realtime tasks the NORMAL scheduler is used and for realtime tasks the REALTIME scheduler is used:

// start logger and communication with non-realtime scheduler
logger_task.start(rt_dt_micro_s,
container::TaskSched::NORMAL);
comm_task.start(rt_dt_micro_s, container::TaskSched::NORMAL);
// start control and ethercat with realtime scheduler,
// attach to isolated CPUs 0 and 1, set priorities to 80
controls_task.start(rt_dt_micro_s,
container::TaskSched::REALTIME, {0}, 80);
ethercat_task.start(rt_dt_micro_s,
container::TaskSched::REALTIME, {1}, 80);

Adjusting timing with secaligned

Each task has a secaligned parameter that can be modified through the MOTORCORTEX Parameter Tree. The secalign parameter aligns the Task’s execution cycle with respect to the system monotonic timer. The secaligned value can be set in the normalized range of [0..1], where 1 corresponds to a full cycle time of the task.

In general, changing secaligned from its default value is not required.

TBD: Diagram & further explanation

Parallelizing execution; splitting the load into Workers

MOTORCORTEX has a special active container to run callable targets asynchronously. This can be used to parallelize heavy computations inside the control blocks or perform blocking calls asynchronously (without blocking the control loop). The advantage of the MOTORCORTEX Worker is that it can be setup to run on a specific CPU with specific priority and scheduler, by default the Worker inherits these properties from the calling task. It is good practice to create the worker in the startOp phase, when the calling task has all its Realtime related properties set.

bool MainControlLoop::startOp_() {
  // creates the Worker with the same priority and CPU affinity as the calling task
  worker1.create(std::string("worker1");
  // creates the Worker with the Normal priority on the non-isolated CPUs
  worker2.create("worker2", mcx::container::TaskSched::NORMAL, mcx::utils::getAvailableCpu());
  return true;
}
bool MainControlLoop::iterateOp_(const container::TaskTime& system_time, container::UserTime* user_time) {
  // moving half of the control loops to the worker
  unsigned int split_the_work = number_of_control_loops / 2;
  for (unsigned int cnt = 0; cnt < split_the_work; cnt++) {
    worker1.start([&, cnt]() {
      punchControl_[cnt].iterate(system_time, user_time);
    });
  }
  // computing the other half in the task
  for (unsigned int cnt = split_the_work; cnt < number_of_control_loops; cnt++) {
    punchControl_[cnt].iterate(system_time, user_time);
  }
  // Important: wait for the worker to finish
  worker1.join(0);
}

Debugging

Cycletime and utilization

Cycle Time and Utilization

MOTORCORTEX Tasks show statistical information in the Parameter Tree that can indicate problems with performance:

  • absolute_cycle_max; maximum cycle time in microseconds since the last reset
  • actual_cycle_max; maximum cycle time over the last second
  • desired_cycle_time_micro_s; the target cycle time in microseconds
  • reset_absolute_cycle_max; if set to 1 the absolute_cycle_max is reset to zero
  • utilization_max; the amount of time the task takes in percentage of the desired_cycle_time_micro_s

Linux debugging tools

There are a number of Linux tools that can deep-dive into your application and the system: