Nightly build steps with Jenkins declarative pipeline

CUDA in Docker

To use CUDA within docker to take advantage of parallel programming on your GPU you need to expose your GPU inside the docker container. To do so you can use the nvidia-docker extension to expose your NVIDIA graphics card and drivers inside the container.

Requirements

NVIDIA Driver 

The first major requirement is to make sure you are using an NVIDIA graphics card and the NVIDIA propriety driver. On Ubuntu you can enable this from Software & Updates:

If using a laptop (or intel chip that includes integrated graphics) you may also need to make sure that you have selected the NVIDIA graphics card as the one in use. Once the driver is installed you can select the card using the NVIDIA X Server Settings applications:

If you had to change either of the above, restart your computer for them to take effect.

You can tell if your NVIDIA card is running by using the nvidia-smi command line tool:

$ nvidia-smi      
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 660M    Off  | 0000:01:00.0     N/A |                  N/A |
| N/A   62C    P0    N/A /  N/A |    236MiB /  1999MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+
$ 

Docker and NVIDIA Docker

Obviously to use docker you must first install it. Follow the instructions on their site to download and install the latest version.

After install docker, you should then install the nvidia docker extension. Installers and instructions are available on the linked github page.

Running nvidia-docker

Once everything is installed you should be able to use the GPU in your container

$ nvidia-docker run -it --rm nvidia/cuda nvidia-smi       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 660M    Off  | 0000:01:00.0     N/A |                  N/A |
| N/A   62C    P0    N/A /  N/A |    236MiB /  1999MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+
$ 

As you can see this is the same as the above output when running the tool outside of the container on my native host.

Choosing Type of OS

There are a number of pre-built images available using different versions of CUDA and popular OS / Containers, including:

• Ubuntu 16.04
• Ubuntu 14.04
• CentOS 7
• CentOS 8

It is also possible to view the dockerfiles used to create these images and extract the required values to re-create your own images. For example, this is the dockerfile for an Ubuntu 16.04 and CUDA 8

Performing nightly build steps with a Jenkinsfile

Note: 2018-12-13: I have a new post with an updated version that works with declarative pipelines. 


Using a Jenkinsfile to control your jenkins builds is an important part of the jenkins 2 workflow for pipeline-as-code. A Jenkinsfile allows you to control what you build, were you build it and all other aspects of your CI flow.

Typically when using pipeline-as-code your build would be triggered by a commit or push from your source control repository. However, there can still be times when you want your build to run on a schedule to perform a long running task e.g. static analysis or a full rebuild of your repository.

Running a nightly build

Jenkins supports running jobs using a trigger which can be controlled with a cron like format. From a Jenkinsfile this can be setup using triggers

  
def triggers = []
triggers << cron('H H(0-2) * * *')
properties (
    [
        pipelineTriggers(triggers)

    ]
)

This will cause your build to trigger sometime between midnight and 2am every day. The above works correctly, however it will cause a build to trigger for every branch in your repository. To limit it to a specific branch you can change it to

def triggers = []
if (env.BRANCH_NAME == "master) {
    triggers << cron('H H(0-2) * * *')
}
properties (
    [
        pipelineTriggers(triggers)

    ]
)

This will limit your scheduled build to only run on the master branch.

Limiting parts of the build to only run at night

Now that you have your build running every night, how do you limit the long running tasks to only trigger from the nightly build?

To do this you must examine the cause of the build. This involves getting the rawBuild data and searching all causes for a particular line in the description. Below is a handy function I've written which can be used to get that information.

// check if the job was started by a timer

// check if the job was started by a timer
@NonCPS
def isJobStartedByTimer() {
    def startedByTimer = false
    try {
        def buildCauses = currentBuild.rawBuild.getCauses()
        for ( buildCause in buildCauses ) {
            if (buildCause != null) {
                def causeDescription = buildCause.getShortDescription()
                echo "shortDescription: ${causeDescription}"
                if (causeDescription.contains("Started by timer")) {
                    startedByTimer = true
                }
            }
        }
    } catch(theError) {
        echo "Error getting build cause"
    }

    return startedByTimer
}

Note: As this is a NonCPS function it must be run outside of a node block.

Note: To get this to work correctly you may have to go to Manage Jenkins > In Process Script Approval, and approve the following signatures

method groovy.lang.Binding getVariables
method hudson.model.Cause getShortDescription
method hudson.model.Run getCause java.lang.Class
method hudson.model.Run getCauses
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild

When I run my build I change my trigger section to

def triggers = []
def startedByTimer = false
if (env.BRANCH_NAME == "master) {
    triggers << cron('H H(0-2) * * *')
    startedByTimer = isJobStartedByTimer()
}
properties (
    [
        pipelineTriggers(triggers)

    ]
)

Then later in my build I can check if the build is a timed build and run the additional analysis checks. For example

if ( startedByTimer ) {
    node("analysis_server") {
        sh script: "make analysis"
    }
}

Implementing Git Flow with Gitlab and Jenkins

As with my last posts I'm going to cover the updates to the build and test systems that I have been making. In my previous posts I covered using CMake, and moving from SVN to Git. In this post I'm going to cover the branching strategy that I implemented after moving to Git. I will cover the branching model chosen and introduce the tools used which allow continuous integration using that model.

Git branching strategies

Git branching strategies are policies on how to use git for development. This allows you to establish a common work-flow that all team members can use and help make updates easier.

There are a number of branching strategies available and they offer various pros and cons depending on your releasing and development methodologies. For our model we have implemented a slight variation on the git flow model. The main changes from this model that we have implemented are:

• master is used as the current development branch. 
• production is used as the release branch. 
• All changes to master and production must be pushed to Gitlab.
• All changes to master and production must be tested via Jenkins.
• All changes to master and production must be code reviewed.

Git flow was chosen as our model because it allows us to release software versions in a consistent and robust manner while still allowing for other developers to continue to work on new features.

Some of the other models such as github flow are more aligned towards software which follows a continuous deployment model where all changes should be pushed to production after testing instead of having to release software to customers.

Other tools

The tools we use to help with our work flow include Gitlab for the central git server and Jenkins for continuous integration.

Gitlab

Gitlab is git repository management tool, which includes user management, code review, merge requests, wiki and more. It could be considered similar to github and is quickly catching up on features and usability. However, the main area where it is ahead of github and what caused me to choose it was that it has an open source community edition which allows for free, easy to install, onsite installations.

Jenkins

Jeninks is a automation server that supports continuous integration and deployment. It is easily extensible and has many plugins to support most common build and integration tools. This allows it to easily integrate with build and source control tools to receive notifications and automatically update, build and test software.

Build System

As previously mentioned this post is about building and testing a software project which is C++ based and uses CMake as the build tool. The core platforms to build for are RedHat based systems including RedHat 5, 6, and 7.

Configuration

In this section I will cover configuration of the various servers. First, I will look at the Gitlab configuration and how the branches and hooks are configured. Secondly, I will cover the Jeninks configuration which builds and tests the software.

Gitlab

The installation of Gitlab is via the Gitlab CE omnibus edition Debian package on an Ubuntu server. This is standard and covered on here.

User accounts, groups and the repository are created. In this example the group is example-group and the project is example-project.

Protected Branches

Two branches master and production are created an set to protected. As described in the Gitlab UI, protect branched are designed to:

• prevent pushed from everybody except masters
• prevent anyone from force pushing to the branch
• prevent anyone from deleting the branch

This means that these branches are sure to exist, only senior developers are allowed to push to them. It also enforces the use of merge requests for code review.

Gitlab Protected Branches

Webhook

Webhooks are configured to sent a HTTP request to the URL http://<jenkins-host>/gitlab/build_now for push events. As described later, this will trigger the Gitlab plugin in Jenkins.

Gitlab webhook

Finally, deploy keys for the Jenkins users are configured on the repository to allow the jenkins user to clone the code.

Jenkins

For this example, Jenkins v1.607 is installed on one server. All builds are performed on the slave nodes buster, earth, and jupiter, where each node has a different version of RedHat installed.

The following are the main plugins used for this example:

• Git Plugin - Allows you to use git as a SCM with Jenkins.
• Gitlab Hook Plugin - Allows Jenkins to receive Gitlab web hooks and trigger builds.

Note: There is a Gitlab Merge Request Builder Plugin but I have not had a chance to configure it yet.

Gitlab Hook Plugin

To configure the Gitlab Hook Plugin go to Manage Jenkins > Configure System and find the section "Gitlab Web Hook"

Gitlab Web Hook configuration

Enable "automatic project creation" and set the project master branch to "master".

Combined with the Gitlab webhook configuration above, this will cause Jenkins to create a new project for every branch that is pushed to Gitlab and have it use the project associated with the master branch as a template.

This allows Jenkins to automatically build and test every commit on every branch that is pushed to Gitlab. By having every branch tested before merging we ensure that all changes are working as expected and that they should be safe to merge into the one of the core branches.

Git Plugin

Configuration of the git plugin is from the project configuration page as shown below:

Git Plugin project configuration

The option to "Clean before checkout" will run a git clean on the project repository to remove any temporary build files from any previous runs of the job.

Job configuration

The configuration of the build job and steps is fairly standard multi-configuration project.

A configuration matrix is configured to run the job on each of the relevant build server slaves.

Build Slave Configuration Matrix

I then created an "Execute Shell" action which will call bash scripts that are in a jenkins folder as part of the repository. This allows you to the actual build commands to be under source control as part of the repository instead of in the Jenkins database. It can also allow slight variations of the build per branch.

Jenkins Execute Shell Build Step

The contents of these scripts can be as as simple or as complex as required. In this example the bash scripts are as follows:

build_step.sh runs CMake and make to compile the software.

#!/bin/bash -ex

mkdir build
cd build
cmake ..
make

test_step.sh runs CTest to make sure all unit tests pass.

#!/bin/bash -ex

cd build
ctest -V

rpm_step.sh uses CPack to create both an RPM and .tar.gz package.

#!/bin/bash -ex
cd build
make package  

Finally after the build has completed one of two options can happens:

• On success the .rpm and .tar.gz build artifacts are archived for use by other jobs.
• On failure an email is sent to the relevant developer group

Post Build Actions

Branch Jobs

All branches will by default create a new job that is a clone of the above master job. These jobs will be called "example-project_<branch name>" and will build the software on every push to Gitlab.

This is also true for the the production branch where a branch "example-project_production" is create. After creation it is possible to make changes to these jobs to add additional build steps required for customer releasable software, for example, you could add a test to make sure that release notes are available or you can SSH the rpm file to a central release server for installation at customer sites.

Merge Requests

As mentioned in the git flow description we have added the step that all branches must be pushed to Gitlab.

The advantage to this is that it requires a merge request to be issued. These merge requests must be approved by another developer to ensure that:

• They meet any coding standards.
• Changes are of sufficient quality.
• Changes are checked by at least 2 developers to help spread knowledge.

Summary

In this post I have show the how to use Gitlab and Jenkins to help implement the git flow branching strategy. This combination allows for continuous integration code review, and helps to enable the building of high quality software.