Saturday, 25 February 2017

It's the end of the line

When working on an install script recently, I came across one of those bugs that make you realise just how pedantic computer programming can be. I had a file that contains a list of yum package names and a script that read the file and did some work on them.

PackageList.txt

redis
python

InstallScript.sh

while read PKG; do
    yum install -y ${PKG}
done < /path/to/PackageList.txt

This file had been working fine as part of our installer for a number of iterations. As part of a developing a new feature I added a new package to the list and saved the file. Thinking this was such a small change that it would just work, I committed it and pushed the changes. However when running the script our tester complained that the new package was missing.

I sat down to debug the issue by checked that the package existed, that the script hadn't changed, and that I had the package name correct. As part of this debugging I resaved the file and it worked again.

After scratching my head, getting a cup of tea and doing some searching, I discovered that the posix standard specifies that a newline character should be added to the end of files. My editor of choice for development is Sublime Text, which by default doesn't add the newline character to the end of the file.

In order to turn it on you should edit your preferences to change the following preference to true.

// Set to true to ensure the last line of the file ends in a 
// newline character when saving
"ensure_newline_at_eof_on_save": true

You may also see the symptom of this issue when committing files to source control and at the end of a diff you will see.

\ No newline at end of file



Friday, 10 February 2017

Inflation Problems

Despite 64-bit operating systems being the default for over 10 years, some of the code I use is still compiled with "-m32" for 32-bit mode. The reasons for this are mostly lack of management will and developer time. As I got time between projects, I decided to update the code so that we can release in both 32-bit and 64-bit mode.

Upgrading the code to be ready for 64-bit mode proved to be a slow task that had many chances for errors. I hope that by showing these errors and some common fixes it helps others to also update their code.

Common Errors

int or unsigned int instead of size_t


On a 32-bit system this isn't really a problem as all 3 types use a 32-bit integer, so you won't get errors. However, it's not portable and on a 64-bit Linux system, size_t is a 64-bit (unsigned) integer. This can cause issues with comparisons and overflow. For example:

string s = "some string";

unsigned int pos = s.find("st");
if( pos == string::npos) {
    // code that can never be hit
}

The above causes issues because string::npos can never be equal to pos as the data type of an unsigned int is too small to match string::npos.


This issue can be caught with the compiler flag -Wtype-limits. Or preferably use -Werror=type=limits to cause the compilation to fail with the following error

error: comparison is always false due to limited range of data type [-Werror=type-limits]

As mentioned this can also cause overflow issues, for example:

unsigned int pos = string::npos;

This causes an overflow because string::npos is too big to fit in a 32-bit integer.

Again this can be caught by a compiler flag, in this case -Woverflow. And again I recommend to use -Werror=overflow to cause a compilation error.

Wrong printf arguments


The logger in our codebase uses printf style formatting for formatting log lines. As a result of this the most common warning on our 64-bit compile was related to this.

The most common cause was related to the above assumption that a size_t is a 32-bit integer. Below is an example of the warning showing this

warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'size_t {aka long unsigned int}' [-Wformat=]
         TRACE(("Insert at position [%u]", pos));

The fix that I used for this warning to use the %zu format specifier for size_t. This was introduced in the C99 standard and should be available in gcc and clang. However, it may not be available in some older versions of the Visual Studio compiler.


TRACE(("Insert at position [%zu]", pos));

I have also seen the above error in relation to other types, for example time_t, uintptr_t, and long. If you are unsure of what the printf argument for a type is, then you can use helpful macros from the C "inttypes.h" header (<cinttypes> if using C++11 or later). This includes macros with the printf arguments for various system typedefs.

Note: Before C++11 you must define __STDC_FORMAT_MACROS before including this header. For example, to print a uintptr_t you can use the macro PRIuPTR


#define __STDC_FORMAT_MACROS 1
#include <inttypes.h>

bool MyList::insert(uintptr_t value)

{
....

    TRACE(("value [%" PRIuPTR "]", value));

Assuming the size of a type is always the same


Again this is somewhat related to the previous points. I saw a number of errors where it was assumed that a particular type was always the same length on different platforms.

The 2 most common were pointers and long.

In our code pointer length issues often manifest as the printf argument error, e.g. using %08x instead of %p but I also saw some cases where a pointer was cast to an int to pass it through a particular function. This would then cause it to then precision on a 64-bit system.


In the case of long it appears that in many cases it was assumed that long was always a 32-bit integer. I came across a number of errors caused by using bitwise operations which assumed that a long was 32-bits. For example:

long offset = getSomeValue();
if ( offset & (1 << 31) )

This causes errors because long is not guaranteed to be a 32-bit integer. If you need to guarantee a size then you should use the correct typedef for that sized integer from the C "stdint.h" header (<cstdint> for C++11). e.g.

#include <stdint.h>

int32_t i32 = get32bitInt();
int64_t i64 = get64bitint();
...

These can then be used in conjunction with the PRIxxx macros from inttypes.h if you need to log / format them

Even with stdint.h there were some ambiguous types that were being cast to / from different types. An example of this was time_t which is not actually defined in a standard. After some googling and testing, I discovered it aligns to the same size as a long (4 bytes on a 32-bit arch, 8 bytes on 64-bit). So when we needed  to pass a time_t value and can't use the time_t typedef I defaulted to using a long.


At the end of the article I show a very simple test program and it's output on RedHat Linux. This shows how the size of types can change depending on compilation mode.

Using the wrong type with malloc

This issue is not actually related to the 64-bit port but the symptoms of it only manifested when we ran the code in 64-bit mode.

There were a couple of blocks of code that were using malloc to get a block of code for an array and these were using the wrong type for the sizeof argument. For example, some code for a hash table included:


typedef struct HT
{
    int num_entries;
    int size;
    HTEntry **table;
} HT;

Then to initialize the table

HT *newtable = NULL;
newtable = (HT*)malloc(sizeof(HT));
newtable->size = size;

newtable->table = (HTEntry**)malloc(sizeof(int)*size);


This has been deployed and run error free for a number of years in our 32-bit software release. However, as the sizeof an int and the size of pointers differ on 64-bit systems, it  caused errors there.

The correct code is:

newtable->table = (HTEntry**)malloc(sizeof(HTEntry*)*size);

Unfortunately I was unable to catch this with any compiler warnings and it caused a crash when run. I had also run some static analyzers over the code which missed this.

Conclusions

The task of updating your code to make it 64-bit compatible is slow, however, can be made easier if you take care to listen to your tools. This includes enabling compiler warnings, making some warnings errors, and using static analysis tools. These will help catch many of the common errors that can occour.


As for the benefit of updating, it will be worth it because:
  •  It will help improve compatibility. As most OSes and software projects are now released in 64-bit mode by default, there is less chance of finding an incompatible package
  • Allow access to new CPU instructions. Compiling with 64bit mode allows access to new instructions and registers. Some initial tests have shown that certain sections of code can be up to 10% faster.
  • Improved code. Keeping the code compiling and working in both environments may lead to more careful programming.

References

http://www.drdobbs.com/cpp/porting-to-64-bit-platforms/226600156?pgno=1

http://www.viva64.com/en/a/0004/

http://www.drdobbs.com/parallel/multiplatform-porting-to-64-bits/184406427

Test program to check common sizes

In order to check sizes, I created a simple test program that will print out the sizes for some common types:

#include 
#include 
#include 
#include 

using namespace std;

int main()
{
    cout << "sizeof(int) : " << sizeof(int) << std::endl;
    cout << "sizeof(unsigned long) : " << sizeof(unsigned long) << std::endl;
    cout << "sizeof(long int) : " << sizeof(long int) << std::endl;
    cout << "sizeof(long long int) : " << sizeof(long long int) << std::endl;
    cout << "sizeof(int32_t) : " << sizeof(int32_t) << std::endl;
    cout << "sizeof(int64_t) : " << sizeof(int64_t) << std::endl;
    cout << "sizeof(double) : " << sizeof(double) << std::endl;
    cout << "sizeof(float) : " << sizeof(float) << std::endl;
    cout << "sizeof(size_t) : " << sizeof(size_t) << std::endl;
    cout << "sizeof(intptr_t) : " << sizeof(intptr_t) << std::endl;
    cout << "sizeof(uintptr_t) : " << sizeof(uintptr_t) << std::endl;
    cout << "sizeof(void*) : " << sizeof(void*) << std::endl;
    cout << "sizeof(char) : " << sizeof(char) << std::endl;
}

To compile and run, you can use:

$> .g++ sizes.cpp -m32 -o t32.sizes
$> ./t32.sizes 
sizeof(int) : 4
sizeof(unsigned long) : 4
sizeof(long int) : 4
sizeof(long long int) : 8
sizeof(int32_t) : 4
sizeof(int64_t) : 8
sizeof(double) : 8
sizeof(float) : 4
sizeof(size_t) : 4
sizeof(intptr_t) : 4
sizeof(uintptr_t) : 4
sizeof(void*) : 4
sizeof(char) : 1



$> .g++ sizes.cpp -o t64.sizes
$> ./t64.sizes 
sizeof(int) : 4
sizeof(unsigned long) :8
sizeof(long int) : 8
sizeof(long long int) : 8
sizeof(int32_t) : 4
sizeof(int64_t) : 8
sizeof(double) : 8
sizeof(float) : 4
sizeof(size_t) : 8
sizeof(intptr_t) : 8
sizeof(uintptr_t) : 8
sizeof(void*) : 8
sizeof(char) : 1


As you can see there are a number of types that have different sizes. These will be the same on all Linux systems, however they aren't guaranteed across different operating systems.

Thursday, 2 February 2017

Build Clang & LLVM tooling on RHEL 7

Clang is a C (and C++) front-end for the LLVM compiler. It provides a fast compiler with really good error messages and great support for writing code analysis and formatting tools. Some of the official tools include:
Third party tools built on top of the clang tooling (and libclang libraries) include:
A good talk by Chandler Carruth about some of the above tools and the future direction for Clang tooling is available on YouTube

Installing Clang

Redhat 7

On RedHat 7, Clang is not included in the official repositories, however older versions (v3.4) are included in the epel repository.

If you are unable to use the epel repository, or want a newer version of clang, the below script can be used to get and install v3.9.1 of llvm, clang, clang tools and the include what you use tool.

mkdir clang_llvm_391_build
cd clang_llvm_391_build
svn co http://llvm.org/svn/llvm-project/llvm/tags/RELEASE_391/final llvm
cd llvm/tools
svn co http://llvm.org/svn/llvm-project/cfe/tags/RELEASE_391/final clang
cd ../..
cd llvm/tools/clang/tools
svn co http://llvm.org/svn/llvm-project/clang-tools-extra/tags/RELEASE_391/final extra
cd ../../../..
cd llvm/projects
svn co http://llvm.org/svn/llvm-project/compiler-rt/tags/RELEASE_391/final compiler-rt
cd ../..
#cd llvm/projects
#svn co http://llvm.org/svn/llvm-project/libcxx/tags/RELEASE_391/final libcxx
#cd ../..
cd llvm/tools/clang/tools
git clone https://github.com/include-what-you-use/include-what-you-use.git
cd include-what-you-use
git checkout clang_3.9
cd ..
echo "" >> CMakeLists.txt
echo "add_subdirectory(include-what-you-use)" >> CMakeLists.txt
cd ../../../..
mkdir llvm.build
cd llvm.build
cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/opt/software/clang -DCMAKE_BUILD_TYPE=Release ../llvm
ninja
mkdir -p /opt/software/clang
cmake -DCMAKE_INSTALL_PREFIX=/opt/software/clang -P cmake_install.cmake


As you can see this installs the software to /opt/software/clang If you want to install to a different location change the CMAKE_INSTALL_PREFIX locations in the line 26 and 29.

The script doesn't compile the version of the C++ standard library (libcxx) available with Clang as I had compiler errors when building it with the default version of gcc (v4.8.5) available with RHEL 7.3

Redhat 6

For RHEL 6, there is also a epel repository with v3.4 available. However, if you want a later version of Clang you have some hoops to run through.

This is because Clang requires a C++11 compiler and Clang v3.9.1, mentioned above, requires at least v4.8 of gcc. The version of gcc available on RHEL 6 is too old and you have to manually build a later version before you can build Clang. You can find instructions on doing so from this blog post.

Using Clang

Compiling

To build your software using Clang with CMake you should override the CMAKE_C_COMPILER and CMAKE_CXX_COMPILER variables. Using the install from my script above this would be done using

$ cmake -DCMAKE_C_COMPILER=/opt/software/clang/bin/clang -DCMAKE_CXX_COMPILER=/opt/software/clang/bin/clang++ ..
$ make

You can see more details in my cmake-examples GitHub repository.

Similar methods of overriding the C and C++ compiler environments may work with other build tools. e.g. using CC and CXX with Makefiles.

Using Clang Static Analyzer

Using the Clang Static Analyzer is easy too as it includes a tool scan-build which can be used to scan your source code at the same time as it builds it

$ /opt/software/clang/bin/scan-build cmake ..
$ /opt/software/clang/bin/scan-build make

On RedHat the above will use gcc to build your software while scanning it with the Clang Static Analyzer.

To get extra coverage for your code I also recommend to use clang to compile it. This can be done at the same time as your static analysis by using the --use-cc and --use-c++ flags for scan-build

$ /opt/software/clang/bin/scan-build --use-cc=/opt/software/clang/bin//clang --use-c++=/opt/software/clang/bin//clang++  cmake ..
$ /opt/software/clang/bin/scan-build --use-cc=/opt/software/clang/bin//clang --use-c++=/opt/software/clang/bin//clang++  make

Advantages of having Clang Available

The main reason I have for using Clang on RedHat is to get access to it's tooling and static analyzer.

However as a side effect of this it also makes the compiler available for use. Using this second compiler can give you more chance of finding errors. For example, when compiling with Clang I had an error:

 file included from /path/to/myclass.cpp:22:
/path/to/logger.h:1:9: warning: '_LIBMYLIB_LOGGER_H_' is used as a header guard here, followed by
      #define of a different macro [-Wheader-guard]
#ifndef _LIBMYLIB_LOGGER_H_
        ^~~~~~~~~~~~~~~~~
/path/to/logger.h:2:9: note: '_LINMYLIB_LOGGER_H_' is defined here; did you mean '_LIBMYLIB_LOGGER_H_'?
#define _LINMYLIB_LOGGER_H_
        ^~~~~~~~~~~~~~~~~
        _LIBMYLIB_LOGGER_H_
6 warnings generated.

This did not cause any errors or warnings on my version of GCC and while it didn't cause any issues (because I only included that header once), it could potentially have lead to a later error.

Sunday, 29 January 2017

Uploading a VirtualBox VM to an Amazon EC2 AMI

There are many blog posts about uploading a VirtualBox VM image to an AWS AMI. However, many are out of date or only cover part of the process. Below I try to describe the current easiest method to convert your custom VM to an AMI.

Reasons for wanting a custom AMI

As part of improving test coverage I am looking into using Amazon EC2 to launch snapshots of test slaves. These snapshots will be partially configured (e.g. basic postgres, redis, and nginx configuration). It would be possible to use a standard Amazon supplied AMI and user data to perform this. However, as I want to have these VMs be as close as possible to our customer installed images, I would prefer to create a local image from our custom install media and then upload it to Amazon.

Initial Investigation

The first step to achieving this is to search on Google. This obviously turned up the AWS documentation which shows that it is possible to upload your own AMIs.  However, it seems to mostly focus on VMware and HyperV. For my purposes, I would much prefer to use a simple VirtualBox image.

After, that I narrowed the search and found a number of blog posts describing the procedure. Many of these describe a convoluted process of creating an image, converting an image to RAW format and then uploading that.

As the AWS documentation describes it being possible to use VMDK and VHD images, and these are supported by VirtualBox, I decided to dig deeper and found that it was possible to upload these if they are created by VirtualBox.

Steps to achieve it

Prepare your VM

Create your VM in VirtualBox using the standard methods. The only specific requirement is that your hard disk file type should be VMDK (Virtual Machine Disk).



In this example, I used a dynamically allocated hard disk file of size 16GB. It should be possible to use a fixed hard disk but that would take longer to upload. For a comparison of how much space you can save, in my simple example, the 16GB dynamically allocated VMDK will be exported as a 570MB OVA file.

Install your VM

The next step is to install the operating system on your OS and configure any required packages. For Linux, this will include an SSH server and network configured as DHCP. For more details, the Amazon documentation describes the main prerequisites and configuration requirements for VM.

Note: Some posts say you have to installing cloud-init onto the OS. This is recommended but I found that it wasn't a hard requirement for my needs.

Export your VM

Not that we have our VM ready for use, we have to export it. In VirtualBox, select File > Export Appliance. And export your virtual machine as below.



You now have a virtual machine in OVA format which is ready for upload to Amazon for use as an AMI.

Upload to Amazon

The steps to upload to Amazon are described in the AWS documentation, but I will repeat the basic steps here for completeness of this post.

Account Permissions

As described here the your AIM user should have the following permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucket",
        "s3:DeleteObject",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": ["arn:aws:s3:::mys3bucket","arn:aws:s3:::mys3bucket/*"]
    }, 
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole",
        "iam:PutRolePolicy"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CancelConversionTask",
        "ec2:CancelExportTask",
        "ec2:CreateImage",
        "ec2:CreateInstanceExportTask",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:DescribeConversionTasks",
        "ec2:DescribeExportTasks",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeInstances",
        "ec2:DescribeTags",
        "ec2:ImportInstance",
        "ec2:ImportVolume",
        "ec2:StartInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances",
        "ec2:ImportImage",
        "ec2:ImportSnapshot",
        "ec2:DescribeImportImageTasks",
        "ec2:DescribeImportSnapshotTasks",
        "ec2:CancelImportTask"
      ],
      "Resource": "*"
    }
  ]
}

Upload your image to S3

Create an S3 bucket (in the same region that you will want to run your EC2 instances in). Then upload the previously exported OVA file to that bucket.

Install the AWS CLI

Install the AWS CLI tools as described here. I used a python virtual environment to keep the tools separate from my standard Ubuntu install.

My steps were:

$ sudo pip install venv
$ mkdir aws
$ cd aws
$ virtualenv awsvenv
$ source awsvenv/bin/activate
(awsvenv)$ pip install awscli

You now have the AWS CLI installed and you need to configure it to work with your AMI user. You can do this by running

(awsvenv)$ aws configure 
AWS Access Key ID: [your access id key]
AWS Secret Access Key: [your access key secret]
Default region name: [your region id e.g. eu-west-1]  
Default output format: [text or json]

Create a VM Import Service Role

You have to create a role to allow you to import your VM and download images from S3. Create a file trust-policy.json

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": { "Service": "vmie.amazonaws.com" },
         "Action": "sts:AssumeRole",
         "Condition": {
            "StringEquals":{
               "sts:Externalid": "vmimport"
            }
         }
      }
   ]
}

Using the AWS CLI create the role vmimport using this file.

 aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json 

Create a file role-policy.json with the following policy, where disk-image-file-bucket is the bucket where the disk images are stored:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource": [
            "arn:aws:s3:::disk-image-file-bucket"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetObject"
         ],
         "Resource": [
            "arn:aws:s3:::disk-image-file-bucket/*"
         ]
      },
      {
         "Effect": "Allow",
         "Action":[
            "ec2:ModifySnapshotAttribute",
            "ec2:CopySnapshot",
            "ec2:RegisterImage",
            "ec2:Describe*"
         ],
         "Resource": "*"
      }
   ]
}

Attach the policy to the role created above.

aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

Import the VM

You are now ready to import the OVA file you uploaded to S3 as an AMI. You can accomplish this using the AWS CLIs import-image command. Create a file containers.json describing the S3 bucket and file.

[
  {
    "Description": "RHEL 7.3 Blog OVA",
    "Format": "ova",
    "UserBucket": {
        "S3Bucket": "my-import-bucket",
        "S3Key": "ami-blogpostvm.ova"
    }
}]

(awsvenv)$ aws ec2 import-image --description "RHEL 7.3 Blog OVA" --disk-containers file://containers.json 
RHEL 7.3 Blog OVA import-ami-fg123456 2 active pending
SNAPSHOTDETAILS 0.0 OVA
USERBUCKET my-import-bucket ami-blogpostvm.ova

Your image is now being imported. This can take a while depending on the size of the image. You can check on the status of the image using the describe-import-image-tasks command and the task id returned from the import-image command

(awsvenv)$ aws ec2 describe-import-image-tasks --import-task-ids import-ami-fg123456
IMPORTIMAGETASKS RHEL 7.3 Blog OVA import-ami-fg123456 2 active pending
SNAPSHOTDETAILS 0.0 OVA
USERBUCKET my-import-bucket ami-blogpostvm.ova

You should see the status of the image going through the following stages
  • pending
  • converting
  • updating
  • active booting
  • active booted
  • active preparing ami
  • complete

(awsvenv)$ aws --output json ec2 describe-import-image-tasks --import-task-ids import-ami-fg123456
{
    "ImportImageTasks": [
        {
            "Description": "RHEL 7.3 Blog OVA",
            "LicenseType": "BYOL",
            "ImageId": "ami-aa123pp",
            "ImportTaskId": "import-ami-fg123456",
            "Status": "completed",
            "Architecture": "x86_64",
            "SnapshotDetails": [
                {
                    "SnapshotId": "snap-0845ad45ad45ad45",
                    "DeviceName": "/dev/sda1",
                    "Format": "VMDK",
                    "DiskImageSize": 570715136.0,
                    "UserBucket": {
                        "S3Key": "ami-blogpostvm.ova",
                        "S3Bucket": "my-import-bucket"
                    }
                }
            ],
            "Platform": "Linux"
        }
    ]
}

Launch your image

Your image has now been uploaded and is available as an AMI. Using the AWS CLI, web interface, or any other tool you can launch a new instance and select your AMI.

Saturday, 1 October 2016

Performing nightly build steps with a Jenkinsfile

Using a Jenkinsfile to control your jenkins builds is an important part of the jenkins 2 workflow for pipeline-as-code. A Jenkinsfile allows you to control what you build, were you build it and all other aspects of your CI flow.

Typically when using pipeline-as-code your build would be triggered by a commit or push from your source control repository. However, there can still be times when you want your build to run on a schedule to perform a long running task e.g. static analysis or a full rebuild of your repository.

Running a nightly build

Jenkins supports running jobs using a trigger which can be controlled with a cron like format. From a Jenkinsfile this can be setup using triggers
  
def triggers = []
triggers << cron('H H(0-2) * * *')
properties (
    [
        pipelineTriggers(triggers)

    ]
)
This will cause your build to trigger sometime between midnight and 2am every day. The above works correctly, however it will cause a build to trigger for every branch in your repository. To limit it to a specific branch you can change it to


def triggers = []
if (env.BRANCH_NAME == "master) {
    triggers << cron('H H(0-2) * * *')
}
properties (
    [
        pipelineTriggers(triggers)

    ]
)
This will limit your scheduled build to only run on the master branch.

Limiting parts of the build to only run at night

Now that you have your build running every night, how do you limit the long running tasks to only trigger from the nightly build?

To do this you must examine the cause of the build. This involves getting the rawBuild data and searching all causes for a particular line in the description. Below is a handy function I've written which can be used to get that information.

// check if the job was started by a timer
// check if the job was started by a timer
@NonCPS
def isJobStartedByTimer() {
    def startedByTimer = false
    try {
        def buildCauses = currentBuild.rawBuild.getCauses()
        for ( buildCause in buildCauses ) {
            if (buildCause != null) {
                def causeDescription = buildCause.getShortDescription()
                echo "shortDescription: ${causeDescription}"
                if (causeDescription.contains("Started by timer")) {
                    startedByTimer = true
                }
            }
        }
    } catch(theError) {
        echo "Error getting build cause"
    }

    return startedByTimer
}

Note: As this is a NonCPS function it must be run outside of a node block.
Note: To get this to work correctly you may have to go to Manage Jenkins > In Process Script Approval, and approve the following signatures

method groovy.lang.Binding getVariables
method hudson.model.Cause getShortDescription
method hudson.model.Run getCause java.lang.Class
method hudson.model.Run getCauses
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild


When I run my build I change my trigger section to

def triggers = []
def startedByTimer = false
if (env.BRANCH_NAME == "master) {
    triggers << cron('H H(0-2) * * *')
    startedByTimer = isJobStartedByTimer()
}
properties (
    [
        pipelineTriggers(triggers)

    ]
)

Then later in my build I can check if the build is a timed build and run the additional analysis checks. For example

if ( startedByTimer ) {
    node("analysis_server") {
        sh script: "make analysis"
    }
}

Thursday, 9 June 2016

Multiple Independent Instances of Gnome Terminal

My typical workflow involves SSHing to multiple servers and switching between them. As a result of this I can often end up having 3+ terminals open into 6+ servers. This results in me often having 15+ terminal windows open on top of my usual browsers, file managers, etc.

I find that it helps me find and sort windows if you can group them based on the server you are logging into instead of the default grouping of all terminals together. To accomplish this grouping you can use a feature in gnome called the window class. This allows you to start applications with a particular WM_CLASS attribute and in the dock, launcher, and <ALT+TAB> menu these windows are grouped together.

In my previous installations of Ubuntu, I had been using either gnome-shell or xfce as the window manager and xterm as my terminal. With this combination I could easily group my terminals and had a handy script to automatically create a menu launcher. However, after upgrading to Ubuntu 16.04 I decided to investigate using gnome-terminal to replace xterm in my workflow.

My first attempt was to just change the above script to launch gnome-terminal instead of xterm (with a slight modification of arguments). I quickly found out that this didn't work and some googling told me that the reason is because gnome-terminal launches a background process called gnome-terminal server which in turn launches and controls the terminal windows. I was able to find a blog on how to launch multiple gnome-terminal-servers, however this required sudo and/or a gnome restart.

After more investigation I found that in Ubuntu /usr/bin/gnome-terminal is a python script that wraps the startup of gnome-terminal and gnome-terminal-server. With a small change to the script, to add a "--class" flag when launching gnome-terminal-server, I was able to fix the issue of terminal windows not showing in multiple groups. The changed script is available from here and the changes are on line 52 and 53 of the script and line 2 and 3 below:

  
        ts = Gio.Subprocess.new(['/usr/lib/gnome-terminal/gnome-terminal-server',
                                 '--class',
                                 name,
                                 '--app-id',
                                 name],
                                Gio.SubprocessFlags.NONE)

Save the script as ~/bin/gnome-terminal-custom and to then launch a terminal in it's own class you can call
  
~/bin/gnome-terminal-custom --disable-factory --app-id com.sshmenu.mylauncher

Wraping up the above in this script to create a desktop menu launcher I can easily launch a new terminal that will automatically SSH into a server and group all terminals for that server together.

Gnome-shell dash showing multiple launchers

gnome-shell dock showing multiple grouped terminal windows

Friday, 8 April 2016

Implementing Git Flow with Gitlab and Jenkins

As with my last posts I'm going to cover the updates to the build and test systems that I have been making. In my previous posts I covered using CMake, and moving from SVN to Git. In this post I'm going to cover the branching strategy that I implemented after moving to Git. I will cover the branching model chosen and introduce the tools used which allow continuous integration using that model.

Git branching strategies

Git branching strategies are policies on how to use git for development. This allows you to establish a common work-flow that all team members can use and help make updates easier.

There are a number of branching strategies available and they offer various pros and cons depending on your releasing and development methodologies. For our model we have implemented a slight variation on the git flow model. The main changes from this model that we have implemented are:
  • master is used as the current development branch. 
  • production is used as the release branch. 
  • All changes to master and production must be pushed to Gitlab.
  • All changes to master and production must be tested via Jenkins.
  • All changes to master and production must be code reviewed.
Git flow was chosen as our model because it allows us to release software versions in a consistent and robust manner while still allowing for other developers to continue to work on new features.

Some of the other models such as github flow are more aligned towards software which follows a continuous deployment model where all changes should be pushed to production after testing instead of having to release software to customers.

Other tools

The tools we use to help with our work flow include Gitlab for the central git server and Jenkins for continuous integration.

Gitlab

Gitlab is git repository management tool, which includes user management, code review, merge requests, wiki and more. It could be considered similar to github and is quickly catching up on features and usability. However, the main area where it is ahead of github and what caused me to choose it was that it has an open source community edition which allows for free, easy to install, onsite installations.

Jenkins

Jeninks is a automation server that supports continuous integration and deployment. It is easily extensible and has many plugins to support most common build and integration tools. This allows it to easily integrate with build and source control tools to receive notifications and automatically update, build and test software.

Build System

As previously mentioned this post is about building and testing a software project which is C++ based and uses CMake as the build tool. The core platforms to build for are RedHat based systems including RedHat 5, 6, and 7.

Configuration

In this section I will cover configuration of the various servers. First, I will look at the Gitlab configuration and how the branches and hooks are configured. Secondly, I will cover the Jeninks configuration which builds and tests the software.

Gitlab

The installation of Gitlab is via the Gitlab CE omnibus edition Debian package on an Ubuntu server. This is standard and covered on here.

User accounts, groups and the repository are created. In this example the group is example-group and the project is example-project.

Protected Branches

Two branches master and production are created an set to protected. As described in the Gitlab UI, protect branched are designed to:
  • prevent pushed from everybody except masters
  • prevent anyone from force pushing to the branch
  • prevent anyone from deleting the branch
This means that these branches are sure to exist, only senior developers are allowed to push to them. It also enforces the use of merge requests for code review.

Gitlab Protected Branches

Webhook

Webhooks are configured to sent a HTTP request to the URL http://<jenkins-host>/gitlab/build_now for push events. As described later, this will trigger the Gitlab plugin in Jenkins.

Gitlab webhook

Finally, deploy keys for the Jenkins users are configured on the repository to allow the jenkins user to clone the code.

Jenkins

For this example, Jenkins v1.607 is installed on one server. All builds are performed on the slave nodes buster, earth, and jupiter, where each node has a different version of RedHat installed.

The following are the main plugins used for this example:
  • Git Plugin - Allows you to use git as a SCM with Jenkins.
  • Gitlab Hook Plugin - Allows Jenkins to receive Gitlab web hooks and trigger builds.
Note: There is a Gitlab Merge Request Builder Plugin but I have not had a chance to configure it yet.

Gitlab Hook Plugin

To configure the Gitlab Hook Plugin go to Manage Jenkins > Configure System and find the section "Gitlab Web Hook"


Gitlab Web Hook configuration


Enable "automatic project creation" and set the project master branch to "master".

Combined with the Gitlab webhook configuration above, this will cause Jenkins to create a new project for every branch that is pushed to Gitlab and have it use the project associated with the master branch as a template.

This allows Jenkins to automatically build and test every commit on every branch that is pushed to Gitlab. By having every branch tested before merging we ensure that all changes are working as expected and that they should be safe to merge into the one of the core branches.

Git Plugin

Configuration of the git plugin is from the project configuration page as shown below:

Git Plugin project configuration

The option to "Clean before checkout" will run a git clean on the project repository to remove any temporary build files from any previous runs of the job.

Job configuration

The configuration of the build job and steps is fairly standard multi-configuration project.

A configuration matrix is configured to run the job on each of the relevant build server slaves.

Build Slave Configuration Matrix


I then created an "Execute Shell" action which will call bash scripts that are in a jenkins folder as part of the repository. This allows you to the actual build commands to be under source control as part of the repository instead of in the Jenkins database. It can also allow slight variations of the build per branch.

Jenkins Execute Shell Build Step
The contents of these scripts can be as as simple or as complex as required. In this example the bash scripts are as follows:

build_step.sh runs CMake and make to compile the software.

#!/bin/bash -ex

mkdir build
cd build
cmake ..
make

test_step.sh runs CTest to make sure all unit tests pass.

#!/bin/bash -ex

cd build
ctest -V

rpm_step.sh uses CPack to create both an RPM and .tar.gz package.

#!/bin/bash -ex
cd build
make package  

Finally after the build has completed one of two options can happens:
  • On success the .rpm and .tar.gz build artifacts are archived for use by other jobs.
  • On failure an email is sent to the relevant developer group
Post Build Actions

Branch Jobs

All branches will by default create a new job that is a clone of the above master job. These jobs will be called "example-project_<branch name>" and will build the software on every push to Gitlab.

This is also true for the the production branch where a branch "example-project_production" is create. After creation it is possible to make changes to these jobs to add additional build steps required for customer releasable software, for example, you could add a test to make sure that release notes are available or you can SSH the rpm file to a central release server for installation at customer sites.

Merge Requests

As mentioned in the git flow description we have added the step that all branches must be pushed to Gitlab.

The advantage to this is that it requires a merge request to be issued. These merge requests must be approved by another developer to ensure that:
  • They meet any coding standards.
  • Changes are of sufficient quality.
  • Changes are checked by at least 2 developers to help spread knowledge.

Summary

In this post I have show the how to use Gitlab and Jenkins to help implement the git flow branching strategy. This combination allows for continuous integration code review, and helps to enable the building of high quality software.