Sunday, 6 August 2017

CUDA in Docker

To use CUDA within docker to take advantage of parallel programming on your GPU you need to expose your GPU inside the docker container. To do so you can use the nvidia-docker extension to expose your NVIDIA graphics card and drivers inside the container.

Requirements

NVIDIA Driver 

The first major requirement is to make sure you are using an NVIDIA graphics card and the NVIDIA propriety driver. On Ubuntu you can enable this from Software & Updates:



If using a laptop (or intel chip that includes integrated graphics) you may also need to make sure that you have selected the NVIDIA graphics card as the one in use. Once the driver is installed you can select the card using the NVIDIA X Server Settings applications:



If you had to change either of the above, restart your computer for them to take effect.

You can tell if your NVIDIA card is running by using the nvidia-smi command line tool:

$ nvidia-smi      
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 660M    Off  | 0000:01:00.0     N/A |                  N/A |
| N/A   62C    P0    N/A /  N/A |    236MiB /  1999MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+
$ 

Docker and NVIDIA Docker

Obviously to use docker you must first install it. Follow the instructions on their site to download and install the latest version.

After install docker, you should then install the nvidia docker extension. Installers and instructions are available on the linked github page.

Running nvidia-docker

Once everything is installed you should be able to use the GPU in your container

$ nvidia-docker run -it --rm nvidia/cuda nvidia-smi       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 660M    Off  | 0000:01:00.0     N/A |                  N/A |
| N/A   62C    P0    N/A /  N/A |    236MiB /  1999MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
+-----------------------------------------------------------------------------+
$ 
  
As you can see this is the same as the above output when running the tool outside of the container on my native host.

Choosing Type of OS

There are a number of pre-built images available using different versions of CUDA and popular OS / Containers, including:
  • Ubuntu 16.04
  • Ubuntu 14.04
  • CentOS 7
  • CentOS 8
It is also possible to view the dockerfiles used to create these images and extract the required values to re-create your own images. For example, this is the dockerfile for an Ubuntu 16.04 and CUDA 8


Clang on RHEL 7 via RPM

In a previous post I discussed how to build CLang and LLVM from source on CentOS 7. That approach works but problems with it are:
  1. It takes a long time to recompile for each user.
  2. It results in large install.
As a result of these drawbacks, I wanted to build an RPM that could result in a minimal install. Some investigation showed that the latest RPM I could find for CentOS 7 was on EPEL and was for CLang 3.4. However, fedora does have an RPM for CLang 3.9. Taking the SRPM for this I was able to modify it to make an RPM for CentOS 7.

Preparation

CMake

To start with we need a version of CMake greater than v3.4.3. The easiest way to do this is via the epel repository:

$ yum install epel-release
$ yum install cmake3
$ cmake3 --version
cmake3 version 3.6.3 

Rpmbuild

You need to install rpmbuild:

$ yum install rpmbuild

Spec files

The spec files are provided below and as mentioned are a modified version of the spec files from the fedora LLVM / CLang v3.9.1 SRPM.7.

LLVM

# Components enabled if supported by target architecture:
%ifarch %ix86 x86_64
  %bcond_without gold
%else
  %bcond_with gold
%endif

Name:       llvm
Version:    3.9.1
Release:    1%{?dist}
Summary:    The Low Level Virtual Machine

License:    NCSA
URL:        http://llvm.org
Source0:    http://llvm.org/releases/%{version}/%{name}-%{version}.src.tar.xz

Source100:  llvm-config.h

BuildRequires:  cmake3
BuildRequires:  zlib-devel
BuildRequires:  libffi-devel
BuildRequires:  ncurses-devel
%if %{with gold}
BuildRequires:  binutils-devel
%endif
BuildRequires:  libstdc++-static

Requires:   %{name}-libs%{?_isa} = %{version}-%{release}

%description
LLVM is a compiler infrastructure designed for compile-time, link-time,
runtime, and idle-time optimization of programs from arbitrary programming
languages. The compiler infrastructure includes mirror sets of programming
tools as well as libraries with equivalent functionality.

%package devel
Summary:    Libraries and header files for LLVM
Requires:   %{name}%{?_isa} = %{version}-%{release}
Requires(posttrans): %{_sbindir}/alternatives
Requires(posttrans): %{_sbindir}/alternatives

%description devel
This package contains library and header files needed to develop new native
programs that use the LLVM infrastructure.

%package doc
Summary:    Documentation for LLVM
BuildArch:  noarch
Requires:   %{name} = %{version}-%{release}

%description doc
Documentation for the LLVM compiler infrastructure.

%package libs
Summary:    LLVM shared libraries

%description libs
Shared libraries for the LLVM compiler infrastructure.

%package static
Summary:    LLVM static libraries

%description static
Static libraries for the LLVM compiler infrastructure.

%prep
%autosetup -n %{name}-%{version}.src

%build
mkdir -p _build
cd _build

# force off shared libs as cmake macros turns it on.
%cmake3 .. \
    -DBUILD_SHARED_LIBS:BOOL=OFF \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_SHARED_LINKER_FLAGS="-Wl,-Bsymbolic -static-libstdc++" \
%if 0%{?__isa_bits} == 64
    -DLLVM_LIBDIR_SUFFIX=64 \
%else
    -DLLVM_LIBDIR_SUFFIX= \
%endif
    \
    -DLLVM_TARGETS_TO_BUILD="X86;AMDGPU;PowerPC;NVPTX;SystemZ;AArch64;ARM;Mips;BPF" \
    -DLLVM_ENABLE_LIBCXX:BOOL=OFF \
    -DLLVM_ENABLE_ZLIB:BOOL=ON \
    -DLLVM_ENABLE_FFI:BOOL=ON \
    -DLLVM_ENABLE_RTTI:BOOL=ON \
%if %{with gold}
    -DLLVM_BINUTILS_INCDIR=%{_includedir} \
%endif
    \
    -DLLVM_BUILD_RUNTIME:BOOL=ON \
    \
    -DLLVM_INCLUDE_TOOLS:BOOL=ON \
    -DLLVM_BUILD_TOOLS:BOOL=ON \
    \
    -DLLVM_INCLUDE_TESTS:BOOL=ON \
    -DLLVM_BUILD_TESTS:BOOL=ON \
    \
    -DLLVM_INCLUDE_EXAMPLES:BOOL=ON \
    -DLLVM_BUILD_EXAMPLES:BOOL=OFF \
    \
    -DLLVM_INCLUDE_UTILS:BOOL=ON \
    -DLLVM_INSTALL_UTILS:BOOL=OFF \
    \
    -DLLVM_INCLUDE_DOCS:BOOL=ON \
    -DLLVM_BUILD_DOCS:BOOL=OFF \
    -DLLVM_ENABLE_DOXYGEN:BOOL=OFF \
    \
    -DLLVM_BUILD_LLVM_DYLIB:BOOL=ON \
    -DLLVM_DYLIB_EXPORT_ALL:BOOL=ON \
    -DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
    -DLLVM_BUILD_EXTERNAL_COMPILER_RT:BOOL=ON \
    -DLLVM_INSTALL_TOOLCHAIN_ONLY:BOOL=OFF 

make %{?_smp_mflags}

%install
cd _build
make install DESTDIR=%{buildroot}

# fix multi-lib
mv -v %{buildroot}%{_bindir}/llvm-config{,-%{__isa_bits}}
mv -v %{buildroot}%{_includedir}/llvm/Config/llvm-config{,-%{__isa_bits}}.h
install -m 0644 %{SOURCE100} %{buildroot}%{_includedir}/llvm/Config/llvm-config.h

%check
cd _build
make check-all || :

%post libs -p /sbin/ldconfig
%postun libs -p /sbin/ldconfig

%post devel
%{_sbindir}/update-alternatives --install %{_bindir}/llvm-config llvm-config %{_bindir}/llvm-config-%{__isa_bits} %{__isa_bits}

%postun devel
[ $1 -eq 0 ] && %{_sbindir}/update-alternatives --remove llvm-config %{_bindir}/llvm-config-%{__isa_bits}

%files
%{_bindir}/*
%exclude %{_bindir}/llvm-config-%{__isa_bits}

%files libs
%{_libdir}/BugpointPasses.so
%{_libdir}/LLVMHello.so
%if %{with gold}
%{_libdir}/LLVMgold.so
%endif
%{_libdir}/libLLVM-3.9*.so
%{_libdir}/libLTO.so

%files devel
%{_bindir}/llvm-config-%{__isa_bits}
%{_includedir}/llvm
%{_includedir}/llvm-c
%{_libdir}/libLLVM.so
%{_libdir}/cmake/llvm

%files static
%{_libdir}/*.a

%changelog
* Sun Aug 06 2017 Thom Troy  - 3.9.1-1
- First build - spec a modified version of fedora25 SRPM

Compiler-rt


%ifarch s390 s390x
# only limited set of libs available on s390(x) and the existing ones (stats, ubsan) don't provide debuginfo
%global debug_package %{nil}
%endif

Name:       compiler-rt
Version:    3.9.1
Release:    1%{?dist}
Summary:    LLVM "compiler-rt" runtime libraries

License:    NCSA or MIT
URL:        http://llvm.org
Source0:    http://llvm.org/releases/%{version}/%{name}-%{version}.src.tar.xz

BuildRequires:  cmake3
BuildRequires:  python
BuildRequires:  llvm-devel = %{version}
BuildRequires:  llvm-static = %{version}

%description
The compiler-rt project is a part of the LLVM project. It provides
implementation of the low-level target-specific hooks required by
code generation, sanitizer runtimes and profiling library for code
instrumentation, and Blocks C language extension.

%prep
%setup -q -n %{name}-%{version}.src

%build
mkdir -p _build
cd _build
%cmake3 .. \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    -DLLVM_CONFIG_PATH:FILEPATH=%{_bindir}/llvm-config-%{__isa_bits} \
    \
%if 0%{?__isa_bits} == 64
        -DLLVM_LIBDIR_SUFFIX=64 \
%else
        -DLLVM_LIBDIR_SUFFIX= \
%endif
    -DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF # could be on?

make %{?_smp_mflags}

%install
cd _build
make install DESTDIR=%{buildroot}

# move sanitizer lists to better place
mkdir -p %{buildroot}%{_libdir}/clang/%{version}
for file in asan_blacklist.txt msan_blacklist.txt dfsan_blacklist.txt cfi_blacklist.txt dfsan_abilist.txt; do
    mv -v %{buildroot}%{_prefix}/${file} %{buildroot}%{_libdir}/clang/%{version}/ || :
done

# move sanitizer libs to better place
mkdir -p %{buildroot}%{_libdir}/clang/%{version}/lib
mv -v %{buildroot}%{_prefix}/lib/linux/libclang_rt* %{buildroot}%{_libdir}/clang/%{version}/lib
mkdir -p %{buildroot}%{_libdir}/clang/%{version}/lib/linux/
pushd %{buildroot}%{_libdir}/clang/%{version}/lib
for i in *.a *.syms *.so; do
    ln -s ../$i linux/$i
done

%check
cd _build
#make check-all

%files
%{_includedir}/*
%{_libdir}/clang/%{version}

%changelog
* Fri Jul 14 2017 Thom Troy  - 3.9.1-1
- First build - spec a modified version of fedora25 SRPM


Compiler-rt


Name:       clang
Version:    3.9.1
Release:    1%{?dist}
Summary:    A C language family front-end for LLVM

License:    NCSA
URL:        http://llvm.org
Source0:    http://llvm.org/releases/%{version}/cfe-%{version}.src.tar.xz

Source100:  clang-config.h

BuildRequires:  cmake3
BuildRequires:  llvm-devel = %{version}
BuildRequires:  libxml2-devel
BuildRequires:  llvm-static = %{version}
BuildRequires:  perl-generators
BuildRequires:  ncurses-devel

Requires:   %{name}-libs%{?_isa} = %{version}-%{release}

# clang requires gcc, clang++ requires libstdc++-devel
# - https://bugzilla.redhat.com/show_bug.cgi?id=1021645
# - https://bugzilla.redhat.com/show_bug.cgi?id=1158594
Requires:   libstdc++-devel
Requires:   gcc-c++


%description
clang: noun
    1. A loud, resonant, metallic sound.
    2. The strident call of a crane or goose.
    3. C-language family front-end toolkit.

The goal of the Clang project is to create a new C, C++, Objective C
and Objective C++ front-end for the LLVM compiler. Its tools are built
as libraries and designed to be loosely-coupled and extensible.

%package libs
Summary: Runtime library for clang
Requires: compiler-rt%{?_isa} >= %{version}

%description libs
Runtime library for clang.

%package devel
Summary: Development header files for clang.
Requires: %{name}%{?_isa} = %{version}-%{release}

%description devel
Development header files for clang.

%package analyzer
Summary:    A source code analysis framework
License:    NCSA and MIT
BuildArch:  noarch
Requires:   %{name} = %{version}-%{release}
# not picked up automatically since files are currently not installed in
# standard Python hierarchies yet
Requires:   python

%description analyzer
The Clang Static Analyzer consists of both a source code analysis
framework and a standalone tool that finds bugs in C and Objective-C
programs. The standalone tool is invoked from the command-line, and is
intended to run in tandem with a build of a project or code base.

%prep
%setup -q -n cfe-%{version}.src
%build
mkdir -p _build
cd _build
%cmake3 .. \
    -DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    -DLLVM_CONFIG:FILEPATH=/usr/bin/llvm-config-%{__isa_bits} \
    \
    -DCLANG_ENABLE_ARCMT:BOOL=ON \
    -DCLANG_ENABLE_STATIC_ANALYZER:BOOL=ON \
    -DCLANG_INCLUDE_DOCS:BOOL=ON \
    -DCLANG_INCLUDE_TESTS:BOOL=ON \
    -DCLANG_PLUGIN_SUPPORT:BOOL=ON \
    -DENABLE_LINKER_BUILD_ID:BOOL=ON \
    \
    -DCLANG_BUILD_EXAMPLES:BOOL=OFF \
%if 0%{?__isa_bits} == 64
        -DLLVM_LIBDIR_SUFFIX=64 \
%else
        -DLLVM_LIBDIR_SUFFIX= \
%endif
    -DLIB_SUFFIX=

make %{?_smp_mflags}

%install
cd _build
make install DESTDIR=%{buildroot}

# multilib fix
mv -v %{buildroot}%{_includedir}/clang/Config/config{,-%{__isa_bits}}.h
install -m 0644 %{SOURCE100} %{buildroot}%{_includedir}/clang/Config/config.h

# remove git integration
rm -vf %{buildroot}%{_bindir}/git-clang-format
# remove editor integrations (bbedit, sublime, emacs, vim)
rm -vf %{buildroot}%{_datadir}/clang/clang-format-bbedit.applescript
rm -vf %{buildroot}%{_datadir}/clang/clang-format-sublime.py*
rm -vf %{buildroot}%{_datadir}/clang/clang-format.el
rm -vf %{buildroot}%{_datadir}/clang/clang-format.py*
# remove diff reformatter
rm -vf %{buildroot}%{_datadir}/clang/clang-format-diff.py*

%check
# requires lit.py from LLVM utilities
#cd _build
#make check-all

%files
%{_libdir}/clang/
%{_bindir}/clang*
%{_bindir}/c-index-test

%files libs
%{_libdir}/*.so.*
%{_libdir}/*.so

%files devel
%{_includedir}/clang/
%{_includedir}/clang-c/
%{_libdir}/cmake/
%dir %{_datadir}/clang/

%files analyzer
%{_bindir}/scan-view
%{_bindir}/scan-build
%{_bindir}/scan-build
%{_libexecdir}/ccc-analyzer
%{_libexecdir}/c++-analyzer
%{_datadir}/scan-view/
%{_datadir}/scan-build/
%{_mandir}/man1/scan-build.1.*

%changelog
* Sun Aug 06 2017 Thom Troy  - 3.9.1-1
- First build - spec a modified version of fedora25 SRPM


Include What You Use


Name:           iwyu
Version:        0.7
Release:        1%{?dist}
Summary:        C/C++ source files #include analyzer based on clang

License:        NCSA
Source0:        https://github.com/include-what-you-use/include-what-you-use/archive/clang_3.9.tar.gz

BuildRequires:  cmake3
BuildRequires:  clang-devel >= 3.9
BuildRequires:  llvm-devel
BuildRequires:  llvm-static
BuildRequires:  zlib-devel
# Scripts are Python 2
BuildRequires:  python2-devel
BuildRequires:  ncurses-devel

# Virtual provide the long name
Provides:  include-what-you-use = %{version}-%{release}
Provides:  include-what-you-use%{?_isa} = %{version}-%{release}

ExclusiveArch: %{ix86} x86_64


%description
"Include what you use" means this: for every symbol (type, function, variable,
or macro) that you use in foo.cc (or foo.cpp), either foo.cc or foo.h
should #include a .h file that exports the declaration of that symbol. The
include-what-you-use tool is a program that can be built with the clang
libraries in order to analyze #includes of source files to find
include-what-you-use violations, and suggest fixes for them. 


%prep
%autosetup -n include-what-you-use-clang_3.9


%build
mkdir build
cd build
%cmake3 -DIWYU_LLVM_LIB_PATH=%{_libdir}/llvm -DIWYU_LLVM_INCLUDE_PATH=%{_includedir} ..
%make_build


%install
%make_install -C build
cd %{buildroot}%{_bindir}
ln -s include-what-you-use iwyu
ln -s fix_includes.py fix_includes
ln -s iwyu_tool.py iwyu_tool


%check
# Need to have the clang header's at the correct relative path (see https://github.com/include-what-you-use/include-what-you-use/issues/100 )
ln -s %{_libdir} %{_lib}
cd build
PATH=$PWD:$PATH
ln -s ../fix_includes.py
ln -s ../fix_includes_test.py
ln -s ../iwyu_test_util.py
ln -s ../run_iwyu_tests.py
ln -s ../tests
%{__python2} run_iwyu_tests.py
%{__python2} fix_includes_test.py


%files
%{_bindir}/include-what-you-use
%{_bindir}/iwyu
%{_bindir}/fix_includes
%{_bindir}/fix_includes.py
%{_bindir}/iwyu_tool
%{_bindir}/iwyu_tool.py
%dir %{_datadir}/include-what-you-use
%{_datadir}/include-what-you-use/*.imp


* Sun Aug 06 2017 Thom Troy  - 0.7
- Update to work on centos 7

Building the spec files

To build the spec files run:

$ spectool -g -R SPECS/llvm.spec
$ rpmbuild -ba SPECS/llvm.spec
$ sudo yum install -y RPMS/x86_64/llvm-static-3.9.1-1.el7.centos.x86_64.rpm \
 RPMS/x86_64/llvm-devel-3.9.1-1.el7.centos.x86_64.rpm \
 RPMS/x86_64/llvm-libs-3.9.1-1.el7.centos.x86_64.rpm \
 RPMS/x86_64/llvm-3.9.1-1.el7.centos.x86_64.rpm
$ spectool -g -R SPECS/compiler-rt.spec
$ rpmbuild -ba SPECS/compiler-rt.spec
$ sudo yum install -y RPMS/x86_64/compiler-rt-3.9.1-1.el7.centos.x86_64.rpm
$ spectool -g -R SPECS/clang.spec
$ rpmbuild -ba SPECS/clang.spec
$ sudo yum install -y RPMS/x86_64/clang-3.9.1-1.el7.centos.x86_64.rpm \
 RPMS/x86_64/clang-libs-3.9.1-1.el7.centos.x86_64.rpm \
 RPMS/x86_64/clang-devel-3.9.1-1.el7.centos.x86_64.rpm
$ spectool -g -R SPECS/iwyu.spec
$ rpmbuild -ba SPECS/iwyu.spec

This will result in you having all RPMS and SRPMS needed to install and use CLang and LLVM v3.9.1