CI Overview - dfki-ric/phobos GitHub Wiki

This Wiki explains the usage of the Phobos CI. You'll find the most answers in the following FAQs. For further details see the Configuration page.

[TOC]

Detailed explanations

• Model Processing Pipeline
  • Model definitions
  • Pipeline Definition
• General CI-Test job
• Pipeline to create PDF-Robot-Visualizations

Overview

Automated model creation, testing and maintenance

As simulation models are often developed in parallel with the real robot, changes are oftenly required. On the other hand the simulation models are used for various purposes and with different software. Therefore, several models might have to be maintained. When maintaining all these models manually, there might happen inconsistencies over the model versions and the current state of the real system.

To ensure that all model versions stay synced with the current development state of the real system, we developed "phobos-CI" a Python-based pipeline to derive the different models automatically.

Structure

To perform this task the latest CAD export is stored in a Git-repository. For deriving the different models from this input, YAML-based definitions are setup in another repository. Using the GitLab-CI pipeline we run process, test and finally deploy the derived models (see the following subsection). As these jobs require plenty of repositories to be managed (multiple input repositories, definitions repository, all output repositories), we maintain a daily updated docker image that already includes a complete autoproj-bootstrap with all required repositories; that way the actual pipeline job takes less time pulling updates. (See Fig. above)

Pipeline jobs and there functions

In a basic configuration file, we set up an ordered list of the definition files for the models to derive as well as a definition of the basic model types. The latter includes information of which files, mesh types and tests to use.

During the following pipeline stages the models are processed in the given order. Then the tests are run and finally the models are deployed. While the pipeline runs any logs created are saved to the job artifacts for easier debugging.

Processing Via the definition file an amount of operations can be defined, that shall be performed to derive the model from the input.

In a first step, it can be defined from what base the respective model shall be derived. This includes both a single URDF model and a model resulting from a previous derivation step. In addition, it is also possible to join multiple input models; e.g. to assemble a complex robot from a trunk model and a leg model, where the latter can be reused to be installed at all respective attachment points. If necessary, the parts can also be mirrored.

In the second step, the model may be further modified. This includes operations like:

• Remove joints and links
• Reorient frames
• Modify joint properties
• Edit collision shapes and mesh formats
• Add extended model properties like sensors and collision bitmask using SMURF robot model
• Define submechanisms for the usage with HyRoDyn
• Export additional files like a model version with a floating base joint added before the root, or URDFs that include just parts of the model

Finally, when executing the processing job, each model is derived according to it's selected type and definition file, and passed via the CI-pipeline-artifacts to the testing job.

Testing Depending on the type of the derived model, different tests can be defined. However, no matter the selection, for each model will be tested whether the model is consistent with latest state on the master-branch. That way we make sure that the working state on master will not be overwritten. For when ever any test fails for a newly generated model, it will not be pushed to master but to develop-branch.

Besides this crucial check, any of these following checks can be used to prevent unwanted changes to the model:

• (Mandatory) Check changes in links and joints (rename, remove, change of joint types)
• (Mandatory) Check changes in the directory and file structure (are files disappearing what can lead to errors for the user)
• Comparison of link masses with prev. model
• Comparison of link transformations with prev. model
• Test whether the model is loadable in pyBullet

For HyRoDyn models we can use HyRoDyn to run some additional tests:

• Test whether the model is loadable in HyRoDyn
• Compare total model mass with prev. model
• Compare total COM position with prev. model
• Compare joint torques to maintain position against gravity
• Compare certain link positions with the previous model (checks the zero joint positions)
• Check position symmetry for defined links

Additionally it is possible to set another set of joint positions or the loaded HyRoDyn model to run those tests again for a different pose.

Deploying Depending on the results of the test stage, the models will be deployed respectively. If no test has failed -- which means that they are absolutely consistent with the previous working state -- they will be deployed directly to the master. Depending on the selected tests, this e.g. would be the case for minor changes of meshes or adding links.

In the other case, for failing tests, the newly generated model is deployed to a develop-branch and a merge request is automatically opened including a short test report. That way the model can be tested manually and when all the changes are wanted and turned out as expected the updated model can be merged manually to master.

If needed, the result model can also be deployed to a second repository, that does not necessarily have to be checked out in the autoproj setup. Moreover the deploying job supports Git-Large File Storage for the mesh repositories, to reduce repository sizes.

Docker Image

The phobos-CI runs inside a Docker Image containing the repositories to process and phobos.