tag:blogger.com,1999:blog-9946657937257485832024-03-11T05:05:30.639+00:00Hops to Raw PointersRandom musings about software development mostly related to C++. I might eventually get around to writing about beer too.twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.comBlogger28125tag:blogger.com,1999:blog-994665793725748583.post-86876178299927090262018-12-13T13:14:00.002+00:002018-12-13T13:14:48.680+00:00Nightly build steps with Jenkins declarative pipeline<script type='text/x-markdown'>
Jenkins [declarative pipeline](https://jenkins.io/doc/book/pipeline/syntax/) is a new(ish) pipeline syntax for Jenkins that allows you to write pipeline-as-code in a Jeninsfile. Unlike a traditional pipeline, instead of writing Groovy code, a declarative pipeline uses a custom DSL. A simple example that builds a C++ application is below:
pipeline {
agent any
stages {
stage('Clean Old Builds') {
steps {
sh 'rm -fr _builds'
}
}
stage('Build') {
sh 'cmake -H. -B_builds/default'
sh 'cmake --build _builds/default'
}
stage('Test')
sh 'cmake --build _builds/default --target test'
}
}
}
The above example, will have 3 stages that can run on any build machine.
* Stage 1, `Clean Old Builds`, will remove any old builds.
* Stage 2, `Build`, will run cmake and make.
* Stage 3, `Test`, will run any unit tests.
This can be cleaner and easier to read than the older Groovy based pipeline syntax.
## Nightly Builds
In a [previous post](https://hopstorawpointers.blogspot.com/2016/10/performing-nightly-build-steps-with.html), I discussed how to run nightly builds using Jenkins Groovy pipelines. I tried to use the same `isJobStartedByTimer` function with declarative pipeline, as follows,
def isTimerBuild = isJobStartedByTimer()
pipeline {
...
triggers {
cron(env.BRANCH_NAME == 'master' ? 'H H(0-2) * * *' : '')
}
stages {
...
stage('Nightly') {
when {
expression {
isTimerBuild == true
}
}
steps {
sh 'cmake --build _builds/default --target cppcheck
}
}
}
}
// define isJobStartedByTimer function as before
@NonCPS
def isStartedByTimer() {
....
}
Unfortunately this failed because of an unauthorized error. I was unable to add the required values to the `In Process Script Approval` as they never appeared in the approval window.
## Improved Method
While looking for a solution to the above problem I came across the following [feature request](https://issues.jenkins-ci.org/browse/JENKINS-41272) in Jenkins. This was added in the [pipeline workflow api plugin](https://wiki.jenkins.io/display/JENKINS/Pipeline+Supporting+APIs+Plugin) in v2.22. The core of this change is that it allows access to the build causes via the `currentBuild` [class](https://javadoc.jenkins.io/plugin/workflow-support/org/jenkinsci/plugins/workflow/support/steps/build/RunWrapper.html), instead of the sandboxed `currentBuild.rawBuild` class.
currentBuild.getBuildCauses()
currentBuild.getBuildCauses(String superClassName)
When using these functions you do not have to enable any `In Script` approvals. To get this to work I updated my `isJobStartedByTimer` as follows:
@NonCPS
def isJobStartedByTimer() {
try {
if(currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').size() > 0) {
echo "build started by timer"
return true
}
} catch(theError) {
echo "Error getting build cause: ${theError}"
}
return false
}
Using this updated function, and the example pipeline from earlier, allows me to have build steps that only run at night.
### Alternative Improved Method
In some cases I found that the above improved method caused a `ClassNotFoundException` saying that the `TimerTrigger` class did not exist. An alternative example, that could be used if you have receive this error, is to use the `currentBuild.getBuildCauses()` version of the function. This works as follows:
@NonCPS
def isJobStartedByTimer() {
try {
for ( buildCause in currentBuild.getBuildCauses() ) {
if (buildCause != null) {
if (buildCause.shortDescription.contains("Started by timer")) {
echo "build started by timer"
return true
}
}
}
} catch(theError) {
echo "Error getting build cause: ${theError}"
}
return false
}
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-41109662868913255132018-12-06T09:57:00.004+00:002018-12-06T09:57:40.661+00:00Cppcheck: More Checks<script type='text/x-markdown'>
In my previous posts, I discussed some [errors](https://hopstorawpointers.blogspot.com/2018/11/cppcheck-basic-checks.html) that Cppcheck can help you find. In this post I'm going to look at some more errors that Cppcheck can prevent.
## Memory Leak from realloc
When using `realloc`, a failure to allocate data will return a null pointer. This can lead to a common error where the original data that you tried to reallocate is leaked. For example:
void realloc_issue() {
int* data = (int*)malloc(4 * sizeof(int));
// do something
data = (int*)realloc(data, 8*sizeof(int));
// do something
free(data);
}
In the above code, the original `data` variable is leaked if the call to `realloc` fails. Using Cppcheck you will get the following warning about the code:
[realloc.cpp:4]: (error) Common realloc mistake: 'data' nulled but not freed upon failure
To fix this error you should allow a temporary to take the returned result from `realloc` and only assign it to data if it is valid memory:
int* new_data = (int*)realloc(data, 8*sizeof(int));
if(new_data)
data = new_data;
## Invalidated Iterator
When using stl containers such as `std::vector`, using methods such as `push_back` and `resize` can invalidate iterators. Cppcheck has a check which can help to catch this mistake. For example:
void pb_it() {
std::vector<std::string> v;
v.push_back("first");
std::vector<std::string>::iterator first = v.begin();
for(int i = 0; i < 100; ++i) {
v.push_back("loop");
}
std::cout << *first << "\n";
}
This will result in the following warning:
[push_back.cpp:10]: (error) After push_back(), the iterator 'first' may be invalid.
Note: If you use `auto` for the iterator or a standard algorithm to fill your vector, then Cppcheck will fail to warn about the error. For example, neither of the following will produce a warning:
std::vector<std::string>::iterator first = v.begin();
std::fill_n(std::back_insertor(v), 100, "loop");
std::cout << *first << "\n";
or
auto first = v.begin();
for ...
std::cout << *first << "\n";
## Invalid Operation on File
Using the C library functions for working on a `FILE*`, there is no type safe way at compile time to catch code that reads from a write only file, or writes to a read only file. However, Cppcheck has warnings which can catch this error.
void read_on_write_only() {
FILE* f = fopen("xxxw", "w");
std::array<char, 100> buf;
fread(buf.data(), 1, 100, f);
fclose(f);
}
void write_on_read_only() {
FILE* f = fopen("xxxf", "r");
int x = 0;
fwrite(&x, sizeof(int), 1, f);
fclose(f);
}
In the above code we use `fread` on a file that was opened as write only, and `fwrite` on a file that was opened read only. Cppcheck will output the following warnings:
[file_ops.cpp:4]: (error) Read operation on a file that was opened only for writing.
[file_ops.cpp:11]: (error) Write operation on a file that was opened only for reading.
Note: This doesn't warn with `std::fstream`. For example, If you open a file with `ios_base::in`, and try to write to it then Cppcheck will fail to warn.
## Class Checks
Cppcheck has a number of both style and warning checks that can hint at potential problems with your class design. For example, take this poorly designed class:
#include <string>
class Multipler
{
public:
Multipler(int x) : x_(new int(x))
{
}
~Multipler() = default;
Multipler& operator=(const Multipler& rhs) {
x_ = new int(*rhs.x_);
return *this;
}
int multiply(int y) const { return y * (*x_); }
void name(const std::string& name) { name_ = name; }
private:
int* x_;
std::string name_;
int vx_;
};
On most [compilers](https://godbolt.org/z/2oUTtQ) this will compile with no warnings. However, Cppcheck will output the following warnings:
[class_design.cpp:6]: (warning) Member variable 'Multipler::vx_' is not initialized in the constructor.
[class_design.cpp:10]: (warning) Member variable 'Multipler::vx_' is not assigned a value in 'Multipler::operator='.
[class_design.cpp:10]: (warning) 'operator=' should check for assignment to self to avoid problems with dynamic memory.
[class_design.cpp:6]: (style) Class 'Multipler' does not have a copy constructor which is recommended since it has dynamic memory/resource allocation(s).
[class_design.cpp:6]: (style) Class 'Multipler' has dynamic memory/resource allocation(s). The destructor is explicitly defaulted but the default destructor does not work well. It is recommended to define the destructor.
[class_design.cpp:6]: (style) Class 'Multipler' has a constructor with 1 argument that is not explicit.
[class_design.cpp:11]: (warning) Possible leak in public function. The pointer 'x_' is not deallocated before it is allocated.
These are good checks, however, you need to have style warnings enabled for some of them. If you only use `enable=warning`, Cppcheck will not warn about the fact that you fail to delete `x_` in the destructor.
## Conclusions about Cppcheck
Cppcheck is a good tool to help you write more robust and safe code. However, it is not a silver bullet to prevent errors. As noted in some of my examples, a small change to your code can mean that Cppcheck will fail to catch an error.
I have found that Cppcheck can be very helpful when taking over an old codebase that uses an older style of C++. After enabling Cppcheck, you can often catch errors that have existed in the code for years and the developers have been lucky not to hit (e.g. mistakes in error handlers). If you are using more modern C++ style, Cppcheck might not be up to date with all changes and may miss some warnings. However, I would still recommend to have it running in your CI system to prevent errors from reaching your production code.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-30008931440512766392018-11-29T12:49:00.000+00:002018-11-29T12:49:27.462+00:00Cppcheck: Basic Checks<script type='text/x-markdown'>
In my previous posts, I've given an [intro to Cppcheck](https://hopstorawpointers.blogspot.com/2018/11/intro-to-cppcheck.html) and also described how to [integrate it with CMake](https://hopstorawpointers.blogspot.com/2018/11/integrating-cppcheck-and-cmake.html).
In this post I'm going to start looking at some of the errors that Cppcheck can help you find in your code.
## Memory Leaks
Every C++ developer will have at some point written code which includes a memory leak. With modern C++ we have tools such as `std::unique_ptr` and `std::shared_ptr` to help with resource allocation, however, when using raw pointers (e.g. in legacy code), using a tool such as Cppcheck can help catch errors. For example:
#include <string>
#include <iostream>
std::string* my_to_string(int x) {
return new std::string(to_string(x));
}
void print_ints() {
for(int i = 0; i < 10; ++i) {
std::string* tmp = my_to_string(i);
std::cout << tmp << "\n";
}
}
When run against a [compiler](https://godbolt.org/z/PqzgkB), this code doesn't produce any warnings or errors, even with the all warnings enabled. However, when run against cppcheck, we can see:
$ cppcheck --enable=warning leak.cpp
Checking leak.cpp ...
[leak.cpp:12]: (error) Memory leak: tmp
## Out of Bounds
Out of bounds memory access can cause issues such as a seg fault or leaking of sensitive data. Using C++ check can help to prevent OOB errors from reaching your final release. For example:
#include <array>
#include <iostream>
const int SZ = 5;
std::array<int, SZ> array_oob() {
std::array<int, SZ> arr;
for(int i = 0; i <= SZ; ++i) {
arr[i] = i;
}
return arr;
}
void carray_oob() {
int arr[5];
for(int i = 0; i <= SZ; ++i) {
arr[i] = i;
}
}
int from_outside(int c) {
std::array<int, SZ> arr;
return arr[c];
}
void use_from_ouside() {
from_outside(100);
}
Will produce the following errors when run with Cppcheck:
$ cppcheck --enable=warning oob.cpp
Checking oob.cpp ...
[oob.cpp:9]: (error) Array 'arr[5]' accessed at index 5, which is out of bounds.
[oob.cpp:17]: (error) Array 'arr[5]' accessed at index 5, which is out of bounds.
[oob.cpp:23]: (error) Array 'arr[5]' accessed at index 100, which is out of bounds.
Interestingly using Cppcheck v1.85 the following code does not produce any error:
std::vector<int> vector_oob() {
std::vector<int> arr;
for(int i = 0; i <= SZ; ++i) {
arr[i] = i;
}
return arr;
}
## Use After Move
Cppcheck will detect the incorrect usage of a moved from variable in the following [code](https://godbolt.org/z/7za0-w):
#include <string>
#include <iostream>
void print(std::string&& to_print) {
std::string moved(std::move(to_print));
std::cout << moved << "\n";
}
void use_after_move() {
std::string hello = "Hello";
print(std::move(hello));
hello[0] = 'h';
std::cout << hello << "\n";
}
With the following error:
$ cppcheck --enable=warning uam.cpp
Checking uam.cpp ...
[uam.cpp:13]: (warning) Access of moved variable 'hello'.
[uam.cpp:14]: (warning) Access of moved variable 'hello'.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-53340606236484827552018-11-22T11:08:00.003+00:002018-11-22T11:08:48.362+00:00Integrating Cppcheck and CMake<script type='text/x-markdown'>
In my [previous post](https://hopstorawpointers.blogspot.com/2018/11/intro-to-cppcheck.html), I gave an introduction to the static analysis tool Cppcheck. In this post, I'm going to show how to integrate Cppcheck into the [CMake](https://cmake.org) build system to generate a `cppcheck-analysis` target for the make / ninja build systems.
## CMake
CMake is a meta build system that can be used to generate files for various build systems, including make and ninja. I would recommend to have a basic knowledge of how CMake works before reading this post. For an introduction to CMake, you can read my [CMake tutorial](https://github.com/ttroy50/cmake-examples).
## FindCppcheck
In CMake you can find dependencies using a `find(ModuleName)` command. This commands looks for a file `FindModuleName.cmake` and will parse this file to find the dependency and add it's requirements. For Cppcheck, you can add a `FindCppcheck.cmake` in a folder `cmake/modules` off the root of your project.
You should then add the following to your root `CMakeLists.txt` to tell it where to search for modules:
# Add a custom CMake Modules directory
set(CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/cmake/modules ${CMAKE_MODULE_PATH})
The code for the FindCppcheck.cmake is below
# Locate cppcheck and add a cppcheck-analysis target
#
# This module defines
# CPPCHECK_BIN, where to find cppcheck
#
# To help find the binary you can set CPPCHECK_ROOT_DIR to search a custom path
# Exported argumets include
# CPPCHECK_FOUND, if false, do not try to link to cppcheck --- if (CPPCHECK_FOUND)
#
# CPPCHECK_THREADS_ARG - Number of threads to use (default -j4)
# CPPCHECK_PROJECT_ARG - The project to use (compile_comands.json)
# CPPCHECK_BUILD_DIR_ARG - The build output directory (default - ${PROJECT_BINARY_DIR}/analysis/cppcheck)
# CPPCHECK_ERROR_EXITCODE_ARG - The exit code if an error is found (default --error-exitcode=1)
# CPPCHECK_SUPPRESSIONS - A suppressiosn file to use (defaults to .cppcheck_suppressions)
# CPPCHECK_EXITCODE_SUPPRESSIONS - An exitcode suppressions file to use (defaults to .cppcheck_exitcode_suppressions)
# CPPCHECK_CHECKS_ARGS - The checks to run (defaults to --enable=warning)
# CPPCHECK_OTHER_ARGS - Any other arguments
# CPPCHECK_COMMAND - The full command to run the default cppcheck configuration
# CPPCHECK_EXCLUDES - A list of files or folders to exclude from the scan. Must be the full path
#
# if CPPCHECK_XML_OUTPUT is set to an output file name before calling this. CppCheck will create an xml file with that name
# find the cppcheck binary
# if custom path check there first
if(CPPCHECK_ROOT_DIR)
find_program(CPPCHECK_BIN
NAMES
cppcheck
PATHS
"${CPPCHECK_ROOT_DIR}"
NO_DEFAULT_PATH)
endif()
find_program(CPPCHECK_BIN NAMES cppcheck)
if(CPPCHECK_BIN)
execute_process(COMMAND ${CPPCHECK_BIN} --version
OUTPUT_VARIABLE CPPCHECK_VERSION
ERROR_QUIET
OUTPUT_STRIP_TRAILING_WHITESPACE)
set(CPPCHECK_THREADS_ARG "-j4" CACHE STRING "The number of threads to use")
set(CPPCHECK_PROJECT_ARG "--project=${PROJECT_BINARY_DIR}/compile_commands.json")
set(CPPCHECK_BUILD_DIR_ARG "--cppcheck-build-dir=${PROJECT_BINARY_DIR}/analysis/cppcheck" CACHE STRING "The build directory to use")
# Don't show thise errors
if(EXISTS "${CMAKE_SOURCE_DIR}/.cppcheck_suppressions")
set(CPPCHECK_SUPPRESSIONS "--suppressions-list=${CMAKE_SOURCE_DIR}/.cppcheck_suppressions" CACHE STRING "The suppressions file to use")
else()
set(CPPCHECK_SUPPRESSIONS "" CACHE STRING "The suppressions file to use")
endif()
# Show these errors but don't fail the build
# These are mainly going to be from the "warning" category that is enabled by default later
if(EXISTS "${CMAKE_SOURCE_DIR}/.cppcheck_exitcode_suppressions")
set(CPPCHECK_EXITCODE_SUPPRESSIONS "--exitcode-suppressions=${CMAKE_SOURCE_DIR}/.cppcheck_exitcode_suppressions" CACHE STRING "The exitcode suppressions file to use")
else()
set(CPPCHECK_EXITCODE_SUPPRESSIONS "" CACHE STRING "The exitcode suppressions file to use")
endif()
set(CPPCHECK_ERROR_EXITCODE_ARG "--error-exitcode=1" CACHE STRING "The exitcode to use if an error is found")
set(CPPCHECK_CHECKS_ARGS "--enable=warning" CACHE STRING "Arguments for the checks to run")
set(CPPCHECK_OTHER_ARGS "" CACHE STRING "Other arguments")
set(_CPPCHECK_EXCLUDES)
## set exclude files and folders
foreach(ex ${CPPCHECK_EXCLUDES})
list(APPEND _CPPCHECK_EXCLUDES "-i${ex}")
endforeach(ex)
set(CPPCHECK_ALL_ARGS
${CPPCHECK_THREADS_ARG}
${CPPCHECK_PROJECT_ARG}
${CPPCHECK_BUILD_DIR_ARG}
${CPPCHECK_ERROR_EXITCODE_ARG}
${CPPCHECK_SUPPRESSIONS}
${CPPCHECK_EXITCODE_SUPPRESSIONS}
${CPPCHECK_CHECKS_ARGS}
${CPPCHECK_OTHER_ARGS}
${_CPPCHECK_EXCLUDES}
)
# run cppcheck command with optional xml output for CI system
if(NOT CPPCHECK_XML_OUTPUT)
set(CPPCHECK_COMMAND
${CPPCHECK_BIN}
${CPPCHECK_ALL_ARGS}
)
else()
set(CPPCHECK_COMMAND
${CPPCHECK_BIN}
${CPPCHECK_ALL_ARGS}
--xml
--xml-version=2
2> ${CPPCHECK_XML_OUTPUT})
endif()
endif()
# handle the QUIETLY and REQUIRED arguments and set YAMLCPP_FOUND to TRUE if all listed variables are TRUE
include(FindPackageHandleStandardArgs)
FIND_PACKAGE_HANDLE_STANDARD_ARGS(
CPPCHECK
DEFAULT_MSG
CPPCHECK_BIN)
mark_as_advanced(
CPPCHECK_BIN
CPPCHECK_THREADS_ARG
CPPCHECK_PROJECT_ARG
CPPCHECK_BUILD_DIR_ARG
CPPCHECK_ERROR_EXITCODE_ARG
CPPCHECK_SUPPRESSIONS
CPPCHECK_EXITCODE_SUPPRESSIONS
CPPCHECK_CHECKS_ARGS
CPPCHECK_EXCLUDES
CPPCHECK_OTHER_ARGS)
# If found add a cppcheck-analysis target
if(CPPCHECK_FOUND)
file(MAKE_DIRECTORY ${CMAKE_BINARY_DIR}/analysis/cppcheck)
add_custom_target(cppcheck-analysis
COMMAND ${CPPCHECK_COMMAND})
message("cppcheck found. Use cppccheck-analysis targets to run it")
else()
message("cppcheck not found. No cppccheck-analysis targets")
endif()
This file includes comments to describe the various sections, however, I'll include some common settings that you may wish to override before calling it.
### Checks to enable
To change the level of enabled checks you can use the `CPPCHECK_CHECKS_ARGS` variable. By default this is set to `--enable=warning`. You can override it to any of the supported check levels in Cppcheck.
### Suppression
As discussed in my previous post you can suppress some warnings in Cppcheck using a suppression file. By default the above script will look for a suppressions file in `.cppcheck_suppressions`. If it exists it will add this to the command when calling cppcheck. To override this file you can change the `CPPCHECK_SUPPRESSIONS` argument.
`set(CPPCHECK_SUPPRESSIONS ${PROJECT_ROOT_DIR}/my_suppressions)
### exitcode
When Cppcheck finds errors you can tell it what error code to use when exiting the program. With the above script the default exitcode is `1`. To override this you can use the `CPPCHECK_ERROR_EXITCODE_ARG`. To set it to use the cppcheck default
set(CPPCHECK_ERROR_EXITCODE_ARG "")
### XML output
Cppcheck supports outputting the results via XML. This can be helpful for CI systems to read the output and add a report. To enable XML output you can set the `CPPCHECK_XML_OUTPUT` variable to the file you want to use:
set(CPPCHECK_XML_OUTPUT "${PROJECT_BINARY_DIR}/analysis/cppcheck/cppcheck_analysis.xml")
### Excluding files
If you have vendored third party code in your repository then you may not want to include that in your Cppcheck analysis. To exclude a file or folder you can create a list `CPPCHECK_EXCLUDES`
set(CPPCHECK_EXCLUDES
${CMAKE_SOURCE_DIR}/3rd_party
${CMAKE_BINARY_DIR}/
)
## Calling FindCppcheck
To call cppcheck you might add something similar to the below snipped to your root `CMakeLists.txt`:
set(CPPCHECK_XML_OUTPUT "${PROJECT_BINARY_DIR}/analysis/cppcheck/cppcheck_analysis.xml")
set(CPPCHECK_EXCLUDES
${CMAKE_SOURCE_DIR}/3rd_party
${CMAKE_BINARY_DIR}/
)
find(Cppcheck)
This will then add the target `make cppcheck-analysis` to your build system. By default this target will fail your build if there are any static analysis errors.
## Conclusion
In this post, I've described how you can integrated Cppcheck into your build system using CMake. This allows you to provide a simplified interface for all developers and your CI system to easily call Cppcheck as they would any other build target.
The new `cppcheck-analysis` target will allow for new developers to quickly call Cppcheck and also supports incremental static-analysis to encourage calling it on larger projects.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-83542824808799778482018-11-15T11:53:00.001+00:002018-11-15T11:56:32.805+00:00Intro to CppCheck<script type='text/x-markdown'>
[Cppcheck](https://cppcheck.sourceforge.net) is an open-source static analysis tool for C++. In this post I'm going to describe how to install and run Cppcheck over your C++ code base.
## Static Analysis
Static analysis is the analysis of code without executing it. It can be used to find common programming errors and enforce coding guidelines. Some common benefits of static analysis include:
* Prevent unexpected / undefined behaviour.
* Find security vulnerabilities and make code more secure.
* Make code more maintainable by enforcing coding standards.
The very first static analysis tool that programmers use is their compiler. In my [previous blog posts](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-hidden-overloads.html), I've discussed how using multiple compilers can help catch errors and prevent undefined behaviour. Using multiple compilers is a great first step in detecting issues, however, dedicated static analysis tools can help to prevent more errors as they can look at more detailed and complex code paths to analyse issues.
## Cppcheck
Cppcheck is a standalone static analysis tool that can perform the following checks:
* Undefined Behaviour
* Dead pointers
* Integer Overflow
* Null pointer dereferences
* Out of bounds checking
* Security checking including some common CVE errors
* Coding Standards
It is integrated into may common CI systems, IDEs and programming environments to allow it to be easily run over different code bases.
### Installing
On Linux, most common package managers include a version of Cppcheck, however, this version is often older and out of date. As a result I recommended to install the latest version which can be found from the [Cppcheck website](http://cppcheck.sourceforge.net/#download).
For windows a binary installer is available.
For other platforms, it is easy to build the latest version from source. The only requirements are CMake and a C++11 compatible compiler. To build and install v1.85 (the latest at time or writing), you can run:
$ wget https://github.com/danmar/cppcheck/archive/1.85.tar.gz
$ tar xvf 1.85.tar.gz
$ cd cppcheck-1.85
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
You should then have `cppcheck` available in your path:
$ cppcheck --version
Cppcheck 1.85
### Basic Usage
#### Single File
To run Cppcheck against a single cpp file you can run `cppcheck filename.cpp`. For a simple example, take the following file.
#include <array>
int main() {
const int ex_sz = 5;
std::array<int, ex_sz> ex;
ex[ex_sz] = 1;
return 1;
}
Under GCC and Clang this compiles without any warnings using `-Wall`. However, with Cppcheck I see:
$ cppcheck test.cpp
Checking test.cpp ...
[test.cpp:6]: (error) Array 'ex[5]' accessed at index 5, which is out of bounds.
This simple tests shows us how using Cppcheck as a static analysis tool can save us from an out of bounds error.
#### Header Files
If you have header files, you can use the `-I` flag to tell cppcheck where to search for them.
cppcheck -I inc/ with_header.cpp
#### Recursive Checks
To recursively check all cpp files in a folder you can pass the folder name to Cppcheck.
cppcheck -I inc/ src/
#### Enabling Checks
As mentioned, Cppcheck has a number of different checks available. You can enable or disable checks by using the `--enable` flag.
--enable=<id> Enable additional checks. The available ids are:
* all
Enable all checks. It is recommended to only
use --enable=all when the whole program is
scanned, because this enables unusedFunction.
* warning
Enable warning messages
* style
Enable all coding style checks. All messages
with the severities 'style', 'performance' and
'portability' are enabled.
* performance
Enable performance messages
* portability
Enable portability messages
* information
Enable information messages
* unusedFunction
Check for unused functions. It is recommend
to only enable this when the whole program is
scanned.
* missingInclude
Warn if there are missing includes. For
detailed information, use '--check-config'.
Several ids can be given if you separate them with
commas. See also --std
### Usage for an existing project
As you can see from the above you have to pass details about your source code to Cppcheck in order for it to understand what files to analyse, how to find dependencies, and what compiler flags to use. The easiest way to do this is to use a [compilation database](http://clang.llvm.org/docs/JSONCompilationDatabase.html).
If you are using [CMake](https://cmake.org/) you can create a compilation database using the `-DCMAKE_EXPORT_COMPILE_COMMANDS=ON` flag, or by adding the following to your root `CMakeLists.txt`.
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
After running CMake with this enabled a file `compile_commands.json` will be created. This will look like:
[
{
"directory": "/home/user/development/project",
"command": "/usr/bin/c++ -Iinc -std=c+11 ... -c ../foo/foo.cc",
"file": "../foo/foo.cc"
},
...
{
"directory": "/home/user/development/project",
"command": "/usr/bin/c++ -Iinc -std=c+11 ... -c ../foo/bar.cc",
"file": "../foo/bar.cc"
}
]
Once you have the compilation database available, you can then run Cppcheck as:
cppcheck --enable=all --project=compile_commands.json
## Conclusions
This post showed how to install and run Cppcheck over your project. It also showed a simple example of how using static analysis can help you to detect errors. In future posts, I hope to show more advanced features of Cppcheck and also some common errors that it can help detect.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-29806116732458958932018-11-08T10:40:00.000+00:002018-11-08T10:40:50.620+00:00Warnings Series - Unused Result<script type='text/x-markdown'>
In this series of posts I'm discussing how compiling with multiple compilers and warnings enabled can help to catch errors at development time. In the last post I discussed how to catch [format](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-return-type.html) errors and use compiler specific methods to annotate your code. In this post, I'm going to look at a standards compliant way to annotate your code to help the compiler identify potential errors.
## Background
Starting with C++11, it is possible to add [attribute specifiers](https://en.cppreference.com/w/cpp/language/attributes) to your code. These can give both your users and the compiler hints on how to treat your code. An example of such an attribute is the [deprecated](https://en.cppreference.com/w/cpp/language/attributes/deprecated) attribute to warn that some code (e.g. class, namespace, or function) should no longer be used and may be removed in the future.
## Unused Result
The `unused-result` warning is issued when a function that uses the `[[nodiscard]]` attribute specifier is called and the returned value is not used. This attribute specifier was added to the language in C++17. It can be used to annotate functions, classes and enums.
### nodiscard functions
The `[[nodiscard]]` attribute will encourage the compiler to issue a warning if a function declared nodiscard has it's returned value not captured. For example, the following code will issue an unused result warning.
[[nodiscard]] int* allocate_int()
{
return new int(1);
}
void test() {
allocate_int();
}
In this code, the return value of `allocate_int` is a raw pointer and should always be deleted. By adding the `[[nodiscard]]` attribute, we are telling the compiler to warn users of the function if they ignore the value. An example of such a warning from Clang is:
warning: ignoring return value of function declared with 'nodiscard' attribute [-Wunused-result]
allocate_int();
^
Note: This only checks that the returned value is captured and not that it is used correctly.
### nodiscard classes
The `[[nodiscard]]` attribute can also be added to classes (and enums) to warn when they are returned by value and ignored. For example:
struct [[nodiscard]] use_me { };
use_me do_stuff() {
return use_me{};
}
void test_use_me() {
do_stuff();
}
This will issue a warning that the use of `do_stuff` in `test_use_me` is not using the returned value from that function. This is only applicable when you return the class by value. For example, the following code will not issue a warning:
use_me USE_ME{};
use_me& do_more_stuff() {
return USE_ME;
}
void test_use_me_ref() {
do_more_stuff();
}
### Ignoring nodiscard
If a function is marked `[[nodiscard]]` but for some reason you know you can safely ignore the result, you can use a void cast to ignore the value. In the case of our `do_stuff` function above, we could call it as below to safely ignore the warning:
static_cast<void>(do_stuff());
While C style casts are often discouraged in new code, for a case such as this it can make for easier to read code.
(void)do_stuff();
## Catching the error
### Clang
With newer versions of Clang, you will receive the `unused-result` warning when compiling with C++11, 14 or 17 on all warning levels. With older versions prior to v3.9, you may receive a warning that the `nodiscard` attribute is not known.
### GCC
Similar to Clang, with newer versions of GCC you will receive the `unused-result` warning when compiling with the C++11, 14 or 17 standards on all warning levels. For GCC versions earlier than GCC 7, you will receive a the `unknown-attribute` warning.
### MSVC
On MSVC 2018, you will receive the warning `C4834` when compiling with the `/std:c++latest` flag. For example:
<source>(12): warning C4834: discarding return value of function with 'nodiscard' attribute
<source>(25): warning C4834: discarding return value of function with 'nodiscard' attribute
Earlier versions of MSVC, do not appear to display any related warnings, however, they do ignore the unknown attribute and allow the code to compile.
## Comparing the errors
To view the errors from each compiler you can use [godbolt](https://godbolt.org/z/05B6WN) to compare the output.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-82922124779574543252018-11-01T11:32:00.000+00:002018-11-01T11:32:50.441+00:00Warnings Series - Format<script type='text/x-markdown'>
In this series of posts I'm discussing how compiling with multiple compilers and warnings enabled can help to catch errors. In the last post I discussed how to turn [return type](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-return-type.html) warnings into errors. In this post, I'm going to look at another warning that I often promote to an errors.
## Background
As discussed in my previous posts, you can turn specific warnings into error using the `-Werror=warning-name` command line flag on GCC and Clang. In some cases you can also annotate code in order enable checking it for a warning type.
## Format
The format warning lets you know when you have used the incorrect [format specifier](http://www.cplusplus.com/reference/cstdio/printf/) for the `printf` family of functions. For example:
long i = 4294967295;
printf("%d\n", i);
In the above example, using `%d` instead of `%ld` may print the value `-1` instead of the expected value `4294967295`. For a simple cause of a log statement, the incorrect value might not matter, however, if you are serializing a value to a file it may cause issues for your users.
In another example:
long i = 1;
printf("%s\n", i);
You are attempting to print a number using the string format specifier. This specifier expects a null terminated string and as a result it will cause undefined behaviour which could create security and stability issues.
## Catching the error
### Clang
When compiling with Clang this is a warning by default when using no compiler flags. To promote the warning to an error you can use `-Werror=format`. Below is an example of the warning from Clang:
<source>:23:18: warning: format specifies type 'int' but the argument has type 'int64_t' (aka 'long') [-Wformat]
printf("%d\n", i);
~~ ^
%ld
<source>:24:18: warning: format specifies type 'char *' but the argument has type 'int64_t' (aka 'long') [-Wformat]
printf("%s\n", i);
~~ ^
%ld
2 warnings generated.
### GCC
For GCC the warning is not enabled by default. It is included as part of the `-Wall` flag or can be enabled individually using `-Wformat`. To promote it to an error you can use `-Werror=format`.
Note: From v8.0 of GCC you will receive a warning with no additional compiler flags.
### MSVC
With MSVC the warning for standard printf functions is enabled by default.
## Attributing your own functions
It can sometimes be the case that you want to make your own function which takes a format specifier (e.g. a logging function):
void my_logger(int level, const char* format, ...)
{
if(level >= LOGGING_LEVEL) {
va_list(args);
va_start(args, format);
vfprintf(stderr, format, args);
va_end(args);
}
}
Which you can call as:
my_logger(WARN,"%s\n", i);
In this case the normal format warning won't appear in you code to warn you of errors. If using GCC and Clang, you can add a non-standard [function attribute](https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html) to tell the compiler that a function [takes a format argument](https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-Wformat-3).
void my_logger(int level, const char* format, ...) __attribute__ ((format (printf, 2, 3)));
After adding this you can will receive the following warning warning:
<source>:26:27: warning: format specifies type 'char *' but the argument has type 'int64_t' (aka 'long') [-Wformat]
my_logger(WARN, "%s\n", i);
~~ ^
%ld
Note: This is not available on MSVC, so the addition of this attribute should be hidden behind a conditional compiler flag.
## Comparing the errors
To view the errors from each compiler you can use [godbolt](https://godbolt.org/z/GYlUFH) to compare the output.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-24297819917858881442018-10-25T11:17:00.003+01:002018-10-25T11:17:55.298+01:00Warnings Series - Return Type<script type='text/x-markdown'>
In this series of posts I'm discussing how compiling with multiple compilers can help to catch errors at development time. In the last post I discussed how to detect [out of range errors](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-autological-out-of-range.html). In this post, I'm going to look at turning some warnings into errors. As an example, I'm using the first warning that I promote to an error on every project I work on.
## Background
As discussed in my previous posts, you can enable warnings under GCC and Clang using the `-W` flags. This can turn on a group of warnings, for example using `-Wall`, or an individual warning like `-Wmaybe-uninitialized`. If you want to be stricter, you can promote those warnings to errors that will stop your program from compiling. This can be done by adding `-Werror=` as a compiler flag, e.g. `-Werror=maybe-uninitialized` will make [unitialized](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-sometimes-uninitialized.html) warnings into errors.
## Return Type
All compilers have a warning regarding missing the return statement from a function. Consider the following code:
int* allocate_int() {
int ret = new int(0);
}
int main() {
int* x = allocate_int();
std::cout << *x;
delete x;
}
As you can see if you the function `allocate_int` is missing a return statement, therefore the variable `x` is undefined. This can then introduce strange logic errors that can be different between builds and runs of your program. One of the first times I ran into this error, the symptom of it was that if logging was enabled a variable was `true` and the program worked. If logging was disabled, an error was always returned as the variable was `false`.
Since then, this is always the first warning that I promote to an error on all projects I work on. For new projects, it prevents the error from being introduced. For legacy projects, it helps find and prevent subtle errors. In most legacy projects I've worked on, enabling this error has resulted in a failed compile.
Note: There is no return statement on the main function and this doesn't produce a warning or error. The reason for this is that the C++ standard specifies that `main` will implicitly return 0 if there is no return statement. This implicit return has resulted in papers such as [p1276](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1276r0.html) calling for main to allow void.
## Catching the error
### Clang
When compiling with Clang and this is a warning by default when using no compiler flags. To promote the warning to an error you can use `-Werror=return-type`.
### GCC
For GCC the warning is not enabled by default in older versions of the compiler. It is included as part of the `-Wall` flag or can be enabled individually using `-Wreturn-type`. To promote it to an error you can use `-Werror=return-type`.
Note: From v8.0 of GCC you will receive a warning with no additional compiler flags, however, you must still use `-Werror=return-type` to cause a compilation error.
### MSVC
With MSVC this results in a compilation error by default.
## Comparing the errors
To view the errors from each compiler you can use [godbolt](https://godbolt.org/z/4cedpO) to compare the output.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-75523327308993506612018-10-18T10:34:00.002+01:002018-10-18T10:34:44.930+01:00Warnings Series - autological out of range<script type='text/x-markdown'>
In this series of posts I'm discussing how compiling with alternative compilers can help to catch errors at development time. In the last post I discussed how to detect [uninitialized errors](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-sometimes-uninitialized.html). In this post, I'm going to look at some logic errors related to using incorrect types. The type of error in this post was encountered when consuming a library and ended up in production code.
## Background
For this example, we have a library that returns a result as a `uint8_t`. The values to return come from defines in the header file.
// coin.h
#define COIN_HEAD 0
#define COIN_TAIL 1
#define COIN_ERR 0xFFFF
uint8_t get_coin(const std::string& coin);
## Autological Out of Range
The code to consume the above header file is as follows:
// game.cpp
int main() {
auto c = get_coin("head");
if(c != COIN_ERR) {
std::cout << "valid coin " << c << std::endl;
}
return 0;
}
In the above code, we get the number representation of a coin face based on the string. If the string is valid, it is printed, otherwise we do nothing.
This code compiles with no warnings on GCC with `-Wall`
### Problems with the code
The problem with the above example is that if you pass an unknown string into `get_coin`, the return will convert the `COIN_ERR` definition from `0xFFFF` into `0xFF`. When you then do the comparison `c != COIN_ERR`, it is always true because `c` can never be equal to `0xFFFF`. This is caused because `COIN_ERR` and the return value of `get_coin` are incompatible.
## Catching the error
### Clang
When compiling with Clang and any warning level you will get the following warning:
<source>:35:10: warning: comparison of constant 65535 with expression of type 'unsigned char' is always true [-Wtautological-constant-out-of-range-compare]
if(c != COIN_ERR) {
~ ^ ~~~~~~~~
1 warning generated.
### GCC
As mentioned, on GCC with the compiler flags `-Wall`, there are no errors in the code. Enabling other warnings with `-Wextra` and `-pedantic` fails to trigger a warning.
### MSVC
The warning failed to trigger on MVSC with any warning level.
## Catching early
If we go back and look at the library code that this came from, the definition of `get_coin` is as follows:
// coin.cpp
uint8_t get_coin(const std::string& coin)
{
if(coin == "head") {
return COIN_HEAD;
} else if(coin == "tail") {
return COIN_TAIL;
}
return COIN_ERR;
}
If you recompile the library, this code will produce a warning on GCC, Clang, and MSVC. The Clang version of this warning is:
<source>:29:12: warning: implicit conversion from 'int' to 'uint8_t' (aka 'unsigned char') changes value from 65535 to 255 [-Wconstant-conversion]
return COIN_ERR;
~~~~~~ ^~~~~~~~
<source>:16:18: note: expanded from macro 'COIN_ERR'
#define COIN_ERR 0xFFFF
If the original library author had paid attention to the warnings, they could have correctly caught the incompatibility in the code and fixed it before it reached our final application. This shows one of the reasons to aim for warning free code across multiple compilers.
## Comparing the errors
To view the errors from each compiler you can use [godbolt](https://godbolt.org/z/13T5BG) to compare the output.
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-69462851590091243682018-10-11T16:39:00.002+01:002018-10-11T16:39:32.700+01:00Warnings Series - Sometimes Uninitialized<script type='text/x-markdown'>
In this series of posts I'm discussing how compiling with alternative compilers can help to catch errors at development time. In the last post I discussed how to detect [hidden overloads](https://hopstorawpointers.blogspot.com/2018/10/warnings-series-hidden-overloads.html). In this post, I'm going to look at some recent errors I came across when getting a legacy library to compile under clang.
## Background
The library is an old library that was written with C++98 and was in the style of "C with classes" rather than C++. When written the author followed a single return policy from functions, which was achieved by using a goto cleanup pattern. For a simple example see:
int func(int num)
{
int ret = 0;
if(num < 50)
goto cleanup;
ret = num;
cleanup:
return ret;
}
The error handling was normally activated by the use of macros that would check a condition and if false, log a message, set the return code, and then goto cleanup. This programming style lead to a lot of functions that have the following pattern:
* Declare all variables at the start of the function.
* Check inputs.
* Function logic (with checks).
* Cleanup.
## Uninitialized Variables
All compilers have some form of uninitialized checking in place. For example GCC, Clang and MSVC will all warn about the uninitialized use of mid in the following function:
int test(int num)
{
int mid;
if(num < mid)
return 100;
return 0;
}
## Maybe / Sometimes Uninitialized
However, with more complex use cases some compilers can miss the use of an uninitialized variable. Below is a simplified (and nonsensical) example of triggering such a use case:
// error handling macro
#define check(A, E) \
if ((A)) { \
ret = E; \
goto cleanup; \
}
int* get_mid()
{
return new int(50);
}
int round_100(int num)
{
int* mid;
int ret = 0;
check(num < 0, -1) // make sure input was valid
mid = get_mid(); // get mid point
check(mid, -1) // check mid was allocated
check(num < *mid, 0)
ret = 100;
cleanup:
delete mid;
return ret;
}
As mentioned above, this uses a macro for error checking and the goto-cleanup pattern of error handling. If a number is entered and is less that 0, we return -1. If it is greater than a median value (which we declare as the pointer mid), we return the value of mid. Otherwise we return 100.
Some basic unit tests for this might be created as:
void test_round()
{
assert(round_100(0) == 0);
assert(round_100(45) == 0);
assert(round_100(55) == 100);
assert(round_100(1000) == 100);
}
Using the above code on GCC (with `-Wall`), everything compiles without warning and all tests pass.
### Problem with the code
There are a lot of problems with this code including lack of RAII, use of goto, etc. but for this post I'm only going to look at the code as it currently is without a full refactor.
On the happy path, the code above works fine and does what it is supposed to do. However, if the user enters a number of less than 0 then we go to the cleanup path and attempt to delete the `mid` pointer. This results in us calling delete on uninitialized memory and leads to undefined behaviour.
## Catching the error
### Clang
If you compile the library code with clang (again with `-Wall`) it gets the following warning:
<source>:24:5: warning: variable 'mid' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
check(num < 0, -1);
^~~~~~~~~~~~~~~~~~
<source>:14:9: note: expanded from macro 'check'
if ((A)) { \
^~~
<source>:30:12: note: uninitialized use occurs here
delete mid;
^~~
<source>:24:5: note: remove the 'if' if its condition is always false
check(num < 0, -1);
^~~~~~~~~~~~~~~~~~
<source>:14:5: note: expanded from macro 'check'
if ((A)) { \
^
<source>:21:13: note: initialize the variable 'mid' to silence this warning
int* mid;
^
= nullptr
This warning is long but tells us where the error happens, and even gives a helpful tip on how to fix it by initializing `mid` to `nullptr`.
### GCC
As mentioned, on GCC with the compiler flags `-Wall`, there are no errors in the code. Enabling other warnings with `-Wextra`, `-pedantic`, and even `-Wmaybe-uninitialized` all fail to trigger the warnings.
### MSVC
On MSVC compiling with the `/W4` flag, the warning appears as below:
z:\tmp\example.cpp(30) : warning C4701: potentially uninitialized local variable 'mid' used
z:\tmp\example.cpp(30) : warning C4703: potentially uninitialized local pointer variable 'mid' used
While this isn't as helpful as the clang warning, it does tell us that there is a problem with the code.
## Comparing the errors
To view the errors from each compiler you can use [godbolt](https://godbolt.org/z/guKz51) to compare the output
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-78664865246121001102018-10-08T10:15:00.003+01:002018-10-08T10:15:56.642+01:00Warnings Series - Hidden Overloads<script type='text/x-markdown'>
In previous posts I've described how to build [Clang](https://hopstorawpointers.blogspot.com/2017/02/build-clang-llvm-tooling-on-rhel-7.html) on [RHEL](https://hopstorawpointers.blogspot.com/2017/08/clang-on-rhel-7-via-rpm.html). This allows us to have access to an alternative compiler to help catch errors early. In this and some upcoming blog posts I'm going to go over some of the errors that using multiple compilers has helped me catch.
## Hidden Overloads
Below is some example code which has multiple types of Dice. The base type is a standard 6 sided Dice and it is using inheritance to make a 20 sided Dice.
// library code
struct Dice {
virtual ~Dice() = default;
// Do a "random" roll of the dice
virtual int roll() { return 6; }
virtual int min() { return 1; }
virtual int max() { return 6; }
};
struct TwentySidedDice : public Dice {
virtual ~TwentySidedDice() = default;
// Do multiple random rolls of the dice
virtual int roll(int times) { return times * 3; }
virtual int max() { return 20; }
};
As with all good code, we have some unit tests to make sure everything is working:
// unit tests
void test_dice() {
Dice die;
assert(die.roll() == 6);
assert(die.min() == 1);
assert(die.max() == 6);
}
void test_twenty_sided() {
TwentySidedDice die;
assert(die.roll(4) == 12);
assert(die.min() == 1);
assert(die.max() == 20);
}
The above is compiled without any warnings on GCC (using `-Wall`) and as all unit tests pass, we feel like we can ship the code as a library.
### Problem with the code
If a client attempted to use the above code, they might try something like this:
// client code
int main()
{
auto die = std::make_unique<TwentySidedDice>();
std::cout << "min is " << die->min() << std::endl;
std::cout << "max is " << die->max() << std::endl;
std::cout << "roll is " << die->roll() << std::endl;
return 0;
}
On first look this code seems reasonable and looks like it should work. However, you would see the following error:
<source>: In function 'int main()':
<source>:43:42: error: no matching function for call to 'TwentySidedDice::roll()'
std::cout << "roll is " << die->roll() << std::endl; // no matching function error
^
<source>:16:17: note: candidate: 'virtual int TwentySidedDice::roll(int)'
virtual int roll(int times) { return times * 3; }
^~~~
<source>:16:17: note: candidate expects 1 argument, 0 provided
Compiler returned: 1
The reason for the error is that based on the rules of C++ overload resolution the `int roll(int i);` function in `TwentySidedDice` is hiding the `int roll();` function from `Dice`.
## Catching the error
### Clang
If you compile the library code with clang (again with `-Wall`) it gets the following warning:
<source>:16:17: warning: 'TwentySidedDice::roll' hides overloaded virtual function [-Woverloaded-virtual]
virtual int roll(int times) { return times * 3; }
^
<source>:9:17: note: hidden overloaded virtual function 'Dice::roll' declared here: different number of parameters (0 vs 1)
virtual int roll() { return 6; }
^
1 warning generated.
Compiler returned: 0
This shows the error early and would let a developer know about the potential problem, so that it can be resolved before shipping to users.
### GCC
As mentioned, on GCC with the compiler flags `-Wall`, there are no errors in the code. To enable this warning on GCC you can explicitly add the `-Woverloaded-virtual` flag.
### MSVC
On MSVC compiling with the `/W4` flag, doesn't show any warnings as the `C4264` warning is off by default. To see the error you can enable warning `C4264` or you can see the following using `/Wall:
<source>(16): warning C4263: 'int TwentySidedDice::roll(int)': member function does not override any base class virtual member function
<source>(18): warning C4264: 'int Dice::roll(void)': no override available for virtual member function from base 'Dice'; function is hidden
<source>(9): note: see declaration of 'Dice::roll'
<source>(7): note: see declaration of 'Dice'
## Comparing the errors
To view the errors from each compiler you can use [godbolt](https://godbolt.org/z/ger4cV) to compare the output
</script>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-39556946547655612912018-09-21T23:25:00.001+01:002018-09-21T23:25:08.329+01:00Deploying to PyPi from from travis-ci<script type='text/x-markdown'>
When building a python project, it's very common to want to deploy to [PyPi](https://pypi.org/). To save effort you can have [travis-ci](https://travis-ci.org) automatically deploy for you.
## Insecure Method
The most basic (and insecure) way to do this is to add a `deploy` step to your `.travis.yml` file, that specifies your PyPi username and password.
deploy:
provider: pypi
user: "Your username"
password: "Your password"
This will always deploy the master branch of your project, but unfortunately it exposes your PyPi username and password.
## Secure Method
To secure your password you can use the [travis command line client](https://github.com/travis-ci/travis.rb), to generate an encrypted password and add this to your deploy configuration. To do this, navigate to your repository and run the following:
$ travis encrypt mypassword
Detected repository as me/myproject, is this correct? |yes| y
Please add the following to your .travis.yml file:
secure: "encrypted password"
Pro Tip: You can add it automatically by running with --add.
Add the `secure` section under the password in `.travis.yml` or as mentioned in the output use the `--add` option to do this automatically.
deploy:
provider: pypi
user: "Your username"
password:
secure: "encrypted password"
## Limit to only Tagged Releases
The above method, will deploy your master branch to pypi. However, it's likely you only want tagged releases to be uploaded to pypi. To limit this you can use the `on` section to say if you only want to release tags.
deploy:
provider: pypi
user: "Your username"
password:
secure: "encrypted password"
on:
tags: true
## Using Test PyPi instance
The [test PyPi](https://test.pypi.org/) instance is a playground for testing if your deployment to PyPi would work. If you are testing the above deploys, you may want to first deploy to this test instance to confirm that everything works. To do this you can use the `server` variable to specify that you want to use the test pypi instance.
deploy:
provider: pypi
server: https://test.pypi.org/legacy/ # Remove for deployment to official PyPi repo
user: "Your username"
password:
secure: "encrypted password"
on:
tags: true
This will send your python package to the test pypi server.
Note: Remember to change your encrypted password to the one you use for the test PyPi instance.
Note: This method can also be used for a self hosted PyPi instance.
</script>twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-14063492671250670532018-07-17T23:58:00.001+01:002018-07-19T17:20:00.315+01:00Home Assistant Input Select via Google Home and IFTTT<script type='text/x-markdown'>
I run [home-assistant](https://www.home-assistant.io) to give me central control of many of my smart home items. One of the integrations available via home-assistant is to expose your various tools to [google assistant](https://www.home-assistant.io/components/google_assistant/). Unfortunately because of limitations with what google will allow, this only works with a subset of integrations including lights, switches and climate devices. As a result of this if you want to use voice to control or query other devices you need a different solution. In this post, I will explain how to make an [input select](https://www.home-assistant.io/components/input_select/) device available via Google home and [IFTTT](https://ifttt.com) using an [appdaemon](https://www.home-assistant.io/docs/ecosystem/appdaemon/) script.
## Input Select
An input select allows a list of user defined values to be make available as a drop down list. In this example I will be using an input select that maps the state of the dishes in my dishwasher. To create the input select add the following to your configuration.yaml:
input_select:
dishwasher:
name: Dishwasher State
options:
- clean
- dirty
icon: mdi:cup-water
This lets me set if there are `clean` or `dirty` dishes in the dishwasher, and gives it a specific [material design icon](https://materialdesignicons.com/).
![input select dishwasher](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-LBnLBY6jOq38kVvrXp2Vy9wz61mKKY1MlWeNAfuJV4IagzKc2yn6SzpnQHJ-1sWUdMEGGkd0nOpwPj1_8c8cjqpjVmqw2Bqy5I6CIjFmJF_CFzK2qCgvFcN_gZ1avBUcG-iyYNtAVN0a/s1600/input_select_dishwasher.png "input select dishwasher")
## AppDaemon
AppDaemon allows you to write python scripts which can interact with home-assistant. The interactions can include:
* Notifications about state changes
* Notifications about events
* Change State
* Fire events
* Call services on home-assistant
For this post you can add the following script to your appdaemon folder:
import appdaemon.plugins.hass.hassapi as hass
#
# App to take events from ifttt and change an input_select
#
#
# Args:
#
# input_select: Input select to trigger
# change_event: Name of the event that will trigger it
# say_event: Name of the event that will trigger it
# say_message: Message to say when an event was called
# media_player: The media player to use to post your message
#
class InputSelectChecker(hass.Hass):
def initialize(self):
if 'input_select' in self.args:
self.input_select = self.args["input_select"]
else:
self.log("No input_select specified, doing nothing")
return
if 'change_event' in self.args:
self.change_event = self.args["change_event"]
else:
self.change_event = None
self.log("No change_event specified")
if 'say_event' in self.args:
self.say_event = self.args["say_event"]
else:
self.say_event = None
self.log("No say_event specified")
if self.change_event:
self.change_handle = self.listen_event(self.change_event_cb, event = self.change_event)
self.log("{} handler setup for {}".format(self.change_event, self.input_select))
if self.say_event:
self.say_handle = self.listen_event(self.say_event_cb, event = self.say_event)
self.log("{} handler setup for {}".format(self.say_event, self.input_select))
self.message = self.args['say_message']
self.media_player = self.args['media_player']
def change_event_cb(self, event_name, data, kwargs):
select_state = self.get_state(entity = self.input_select)
self.log("change event select state = {}".format(select_state))
self.log(self.get_state()[self.input_select]['attributes'])
all_options = self.get_state()[self.input_select]['attributes']['options']
self.log("{}".format(all_options))
if 'option' in data:
self.log("Change to {}".format(data))
change_to = data['option'].lower()
for o in all_options:
if change_to == o.lower():
self.select_option(self.input_select, o)
return
self.log("Invalid option {}".format(data))
def say_event_cb(self, event_name, data, kwargs):
select_state = self.get_state(entity = self.input_select)
self.log("say select state for {} = {}".format(self.input_select, select_state))
self.log(self.message)
self.call_service("tts/google_say", entity_id = self.media_player,
message = self.message.format(select_state=select_state))
This script will allow you to listen for configurable events and then either change the configured input select option, or say a TTS message over a configured media player. The message must be in the following format `optional start of message {select_state} optional rest of message`, where `{select_state}` is where you want the state of your input select to be added to the message.
Note: If you want to use a different TTS service, then you can rename it in the script or make it available as an argument.
To configure this add the following to your `apps.yaml`:
dishwasher_state:
module: ifttt_input_select_handler
class: InputSelectChecker
input_select: input_select.dishwasher
change_event: CHANGE_INPUT_SELECT_DISHWASHER
say_event: SAY_INPUT_SELECT_DISHWASHER
media_player: media_player.kitchen
say_message: 'The dishwasher has {select_state} dishes'
If you only want to support one of either "change status", or "say status", then you should only configure the event for that handler.
## IFTTT
If this then that (IFTTT) allows you to receive an event from one service and then call another service. In this example I make 2 IFTTT applets.
Each applet will receive an event from the [Google assistant](https://ifttt.com/services/google_assistant) service and then call a [web-hook](https://ifttt.com/services/maker_webhooks) to call home-assistant.
The first one to get the status is `If you say "What's the status of the dishwasher?" Then "make a web request"`
The second to change the status is `If you say "Change the dishwasher to $" Then "make a web request"`
### Say status applet
Screenshots of creating the say status applet are below:
![ifttt step 0](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix2-Chs3mXAQ3KiAH6T9dciTdQU9lHwvuGw93Xxsev-cEsQZZxBFM_eE9f9aOBb0MUIr3l_uX90yhfqq0mkj8e8rppMcvjc2zZgLo72eTrTsqvfet_cfdwffpGJg9Eagg5IZwYOXUhppOe/s800/step0.png "ifttt step 0")
Step 0: Select `this`.
![ifttt step 1](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDDX8_U2EegDzNhTZknqavI9SY1b5mJlb28j2aAEpF1ozB8i_DuFlEHzlBhGnnfdjW33Z7Q-t6gKCsoHAUoiplxdzCgK0i3iEvU4foQugMiqVjXfdy18pN_N6ZXaMQfs7WuoSxiNubmgmn/s800/step1.png "ifttt step 1")
Step 1: Choose `Say a simple phrase` as the trigger.
![ifttt step 2](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPQAGf2Pawe7CsD_Nrd0kxELD00rn-kxw5wGYHSaUwCH8-9Jk47glLckE2Nhyk5klpuorTXIz-guSWT5ExTfRcc3S3cgXmJUKTA20FphDIeJpUBaCSxCNV18nmEqh6IYE2fkGPALyeivj-/s800/step2.png "ifttt step 2")
Step 2: Choose your trigger phrases and what will be the response from Google.
![ifttt step 3](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvJGr72weAlLVf6L98YWxtl2T3lP_8du5_m3Hl77ksNbFUwoRGksQ88RR3zvx_dv3U1_A9lXLgJy5zhixfVihwbsEXNDj9mfNSR0uh910PgkVXJx92sal2r1-sPRYZon4xVHbVB2m9qKbW/s800/step3.png "ifttt step 3")
Step 3: Choose `that`.
![ifttt step 4](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsSZUsbr3_WdoMStswzj5QKq9bMUonHJz8RcPPtWn8Gx34fnQOjHvglYx612q9vXDpUwWB-RVq_hE0o3MH6xnDj9oBPimPJ_PqOSsAEx01Ta3xcTT42ouW8YfYEDSwOBvFGvDw6oqmcB1V/s800/step4.png "ifttt step 4")
Step 4: Setup the web-hook to call your home assistant instance. This will trigger the event `SAY_INPUT_SELECT_DISHWASHER`, which is the event we configured when setting up our appdaemon app.
### Change status applet
The change status applet is very similar to the above. The differences are:
* In step 1, choose `say a phrase with a text ingredient`
* In step 2, the trigger phrases must include a `$` in the place where you want to have our custom input. For example `Change the dishwasher to $`, where $ should be `clean` or `dirty`.
* In step 2, set the response to `Setting the dishwasher to $`
* In step 4, for the web-hook will be to `https://[HA_URL]/api/events/CHANGE_INPUT_SELECT_DISHWASHER` and the post data will be `{"option": "{{TextField}}"}`.
These changes will allow you to send your text back to home-assistant and if that matches your input_select option it will be selected.
## Conversation
The conversation you have with Google are below.
Change Status:
User: "Hey Google, Change the dishwasher to dirty"
Google: "Setting the dishwasher to dirty"
Check Status:
User: "Hey Google, what is the status of the dishwasher?"
Google: "Checking"
Google: "The dishwasher has dirty dishes"
### Limitations
The limitations of this are:
* You have to setup the IFTTT trigger and appdaemon app for each input select in home assistant.
* When checking the status there is a 2 stage response. First saying "Checking" and then maybe 2 seconds later saying the status response.
* You must manually set the speaker that you respond to.
</script>twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-42419239512710734952018-06-29T20:03:00.000+01:002018-06-29T20:03:29.516+01:00pybind11 and python sub-modules<script type='text/x-markdown'>
In my [last post](https://hopstorawpointers.blogspot.com/2018/06/using-c-code-from-python-with-pybind11.html), I introduced pybind11 and some basic examples. In this post I want to show how to use python sub-modules with
your exported bindings. In particular, I want to show how we structured our bindings in sub-modules when the C++ code was in different
libraries in our main project.
## Python sub-module
A python sub-module is accessible from python like:
>>> from ork import peon
>>> peon.work_work()
I'm not that kind of Orc
In this example, `ork` is the main module and `peon` is the sub-module. In pure Python these might have the folder structure
ork
__init__.py
peon
__init__.py
## C++ Layout
For our code, we had a project that has multiple C++ libraries and we wanted to expose bindings from each library as a
different sub-module of `ork` in Python:
ork
CMakeLists.txt
peon
CMakeLists.txt
include/
src/
grunt
CMakeLists.txt
include/
src/
warlock
CMakeLists.txt
include/
src/
So we wanted to export some functionality from `ork.peon`, `ork.grunt`, and `ork.warlock` to python.
## Exporting the bindings
### Single Shared Object
The easiest way to do this is to create a single shared object using pybind11. This will include all
exported bindings for all the libraries you want to export.
A simplified example would be to add a new `ork_bindings.cpp` after your other libraries:
PYBIND11_MODULE(orc, mymodule) {
mymodule.doc() = "Orks live here";
py::module peon = mymodule.def_submodule("peon", "A peon is a submodule of 'ork'");
peon.def("work_work", &Peon::work_work, "Do some work");
py::module grunt = mymodule.def_submodule("grunt", "A grunt is a submodule of 'ork'");
grunt.def("work_work", &Grunt::work_work, "Do some work");
py::module warlock = mymodule.def_submodule("warlock", "A warlock is a submodule of 'ork'");
warlock.def("work_work", &Warlock::work_work, "Do some work");
}
In your `CMakeLists.txt` you export the the module as:
pybind11_add_module(ork, ork_bindings.cpp)
target_add_library(ork
PRIVATE
peon
grunt
warlock
)
This works fine for a small amount of libraries and exported code. However, I didn't like this approach as it moved your export
code away from your main code. This would make it easy to forget to add a new function to a binding and allow for consistency issues to creep into the project.
### Multiple Shared Objects
Our chosen approach was to use a separate bindings library for each C++ library to be exported as a sub-module. Then use a Python module as the main module.
To do this we added the following to our code:
ork/
CMakeLists.txt
peon/
CMakeLists.txt
include/
src/
bindings.cpp
grunt
CMakeLists.txt
include/
src/
bindings.cpp
warlock
CMakeLists.txt
include/
src/
bindings.cpp
An example `bindings.cpp` for peon is:
PYBIND11_MODULE(pypeon, m)
{
// rename to a submodule
m.attr("__name__") = "ork.pypeon";
m.doc() = "A peon is a submodule of 'ork'";
m.def("work_work", &Peon::work_work, "Do some work");
}
These bindings were added in each `CMakeLists.txt` as:
pybind11_add_module(pypeon, src/bindings.cpp)
target_link_library(pypeon
PRIVATE
peon
)
When installing your library you then install as:
ork/
__init__.py
pypeon.[python_info].so
pygrunt.[python_info].so
pywarlock.[python_info].so
One of the main problems with this approach is the naming of the sub-modules. As the C++ libraries are called
`peon`, `grunt`, and `warlock`, it is not possible to have another CMake target with the same name. Therefore, you have to
have a slightly different name. In the above example, I have chosen to add `py` as a prefix for the sub-module names.
## Conclusions
So far we have found our approach to work, even with the downside of having a prefix to the name. This allows us to make sure
bindings code lives as close a possible to our C++ code and we can conditionally choose which modules to export using CMake options.
</script>twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-30378412561196991902018-06-26T23:40:00.000+01:002018-06-27T14:25:00.524+01:00Using C++ code from Python with pybind11<script type='text/x-markdown'>
I have recently been working on a large C++ code base and needed to make certain parts of the code available in Python for use by our other teams. We looked at a number of different frameworks for this and eventually decided to go with [pybind11](http://github.com/pybind11). In this blog post I will describe the basic usage of pybind11 and how to call your exported code from python. For a full look at the code check out my example repository from [here](http://github.com/ttroy50/pybind11-example)
## Basic Example
The examples below can be considered part of a single library that is being made available. An example cpp file is available [here](https://github.com/ttroy50/pybind11-example/blob/master/bindings.cpp)
### Method
Consider the following method that you want to make available to python
int add(int x, int y) {
return x + y;
}
To make this available to python you add the following binding:
PYBIND11_MODULE(pybindings, mymodule) {
using namespace pybind11::literals; // for _a literal to define arguments
mymodule.doc() = "example module to export code";
mymodule.def("add",
&add,
"Add 2 numbers together",
"x"_a,
"y"_a);
}
To explain the above a little bit we have
`PYBIND11_MODULE(pybindings, m)` - This defines the module name pybindings, with the variable `mymodule` used to reference it.
`mymodule.doc()` - Create a doc string for the module which will be displayed when using the `help` function in python.
`mymodule.def` - Define the function you want to export to python. The arguments to this are the name of the function in python, The C++ function to export, The doc string for the function, Any arguments for the function
Once build this function could be used as:
>>> import pybindings
>>> pybindings.add(1, 2)
3
### Class
Consider the class:
class Adder {
public:
Adder(int x) : x_(x) {};
int add(int y) {
return x_ + y;
}
void setAddition(int x) {
x_ = x;
}
int getAddition() {
return x_;
}
private:
int x_;
};
To export this, under your `PYBIND11_MODULE` add the following:
pybind11::class_<Adder>(mymodule, "Adder")
.def(pybind11::init<int>())
.def("add", &Adder::add)
.def_property("addition", &Adder::getAddition, &Adder::setAddition);
This is similar to the method example above except we now define the class using `pybind11::class_`. The class type is passed as a template argument to this and functions are defined as part of the class instead of the module.
The constructor is defined by calling `pybind11::init`, and you can add access to class variables as properties using the `def_property` function.
Once built it can be run using:
>>> import pybindings
>>> a = pybindings.Adder(1)
>>> a.add(2)
3
>>> a.addition = 4
>>> a.add(2)
6
### stl
pybind11 seamlessly supports using STL containers within python. To to this you import the `pybind11/stl.h` header to make the bindings available.
std::string join(std::vector<char> tojoin) {
std::string ret;
ret.reserve(tojoin.size())
for(auto c: tojoin) {
ret += c;
}
}
To export add the function definition to your module:
mymodule.def("join",
&join,
"Join a list of strings",
"tojoin"_a);
This is the very similar to the `add` function from our first example and shows how pybind11 seamlessly supports STL containers.
To call this from python:
>>> import pybindings
>>> s = pybindings.join(['a', 'b', 'c'])
>>> print(s)
abc
>>> type(s)
<class 'str'>
## Building the Example
Pybind11 has good support for CMake and be easily integrated into a CMake project. To do this add the following to your `CMakeLists.txt`
find_package(pybind11)
pybind11_add_module(pybindings bindings.cpp)
This will add a target to build a library `pybindings.[python-information].so`.
> NOTE: The name of the library must add match the name of the module you defined in your C++ code.
## Using the binding
To use the bindings you add the location of the above shared object into your PYTHONPATH. Assuming your built your code in `/data/code/build` and have a python file `/data/code/bindings.py` you can run:
PYTHONPATH=/data/code/build/ python3 /data/code/bindings.py
The source of bindings.py can be found [here](https://github.com/ttroy50/pybind11-example/blob/master/bindings.py)
</script>twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-2365895577082032182018-04-18T22:49:00.001+01:002018-04-18T22:49:55.532+01:00Rate LimitingAt work I was recently tasked with integrating a rate limiting solution into out API. The purpose of the rate limiter was to cap the number of requests a user can make against an API over a specific period of time. This can help:<br />
<br />
<ul>
<li>Prevent spam attacks.</li>
<li>Stop spikes in traffic from overloading our servers and degrading performance.</li>
<li>Help identify and stop misbehaving clients.</li>
</ul>
<br />
A user in the context of rate limiting is any entity that you wish to limit. Users can be identified by any unique identifier. Some examples include:<br />
<ul>
<li>IP Address </li>
<li>User ID</li>
<li>Device Identifier </li>
<li>Customer Name (for B2B customers)</li>
<li>URL. E.g. only X number of POST requests to /withdrawl URL</li>
</ul>
Or any combination of the above.<br />
<br />
The main requirement for the solution was that our front end services can run on multiple containers in a distributed manner so our rate limiting must work across servers. This would involve using an external key-value store (redis) to store the rate limiting information. As a result of this many of the details described below are directly related to how redis works. These details can normally be applied to other cache services as required.<br />
<br />
For the remainder of this blog post I will discuss the various rate limiting algorithms I investigated, our chosen algorithm and some additional implementation details. <br />
<br />
<h2>
Rate Limiting Algorithms</h2>
<h3>
Fixed Window Counters</h3>
Fixed window counters are the simplest method of rate limiting. They work by maintaining a counter with the number of times an entity has hit the endpoint within a fixed window. If the user makes more than the allocated number of requests, then new requests are rejected. At the start of the new window the counter is reset to 0 and the user may make additional requests. A window is typically associated with a time and the counter is reset at the start of the period. Some examples include:<br />
<ul>
<li>Day window: Reset at the same time every day, e.g. midnight</li>
<li>Hour window: Reset on the hour, e.g. 12:00:00, 13:00:00 ...</li>
<li>15 minute window: Reset every 15 minutes e.g. 12:00:00, 12:15:00 ...</li>
</ul>
The key for the cache service would be <span style="font-family: "courier new" , "courier" , monospace;">{id}_{windowTimestamp}</span>, where id could be the user id of the user and windowTimestamp would be the timestamp at the start of the window.<br />
<br />
In the following example the user is allowed to make 3 requests in a 60 second window.<br />
<br />
<table class="relative-table wrapped confluenceTable tablesorter tablesorter-default stickyTableHeaders" role="grid" style="-webkit-text-stroke-width: 0px; background-color: white; border-collapse: collapse; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Noto Sans", Ubuntu, "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; margin: 0px; orphans: 2; overflow-x: auto; padding: 0px; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; width: 537px; word-spacing: 0px;"><colgroup><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col></colgroup><colgroup><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col></colgroup><thead style="left: 410px; margin-top: 0px; position: static; top: 77px; width: 591px; z-index: 3;">
<tr role="row"><th data-column="0" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; max-width: none; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div style="color: #333333; margin: 0px; padding: 0px;">
Time</div>
</th><th data-column="1" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; max-width: none; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div style="color: #333333; margin: 0px; padding: 0px;">
Action</div>
</th><th data-column="2" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; max-width: none; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Counter Value</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="3" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; max-width: none; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Additional</div>
</th></tr>
</thead><tbody>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:05</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120000: 0 → 1</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">Key is user1_1515120000</td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:15</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120000: 1 → 2</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:01</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120100: 0 → 1</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Counter is reset as the 60 second period is is over. New key is user1_1515120100</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:10</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120100: 1 → 2</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:40</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120100: 2 → 3</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:50</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120100: 3 → 4</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">User is rejected as they are over limit</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:02:20</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120200: 0 → 1</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">Allowed as counter reset and new key is user1_1515120200</td></tr>
</tbody></table>
<br />
Note: Old keys can be set to automatically expire a fixed period after they are done. This may require 2 calls to redis to have an INCR, then EXPIRE command called. <br />
<br />
<h4>
Pros:</h4>
The advantages of this approach are:<br />
<ul>
<li>Simple to implement</li>
<li>INCR is an atomic redis command. </li>
<li>Low memory requirement.</li>
</ul>
<h4>
Cons:</h4>
<ul>
<li>It can allow a user to go above their allowed quota in a rolling 60 second period. For example, in the above limit of 3 requests per 60 seconds, if the user made 3 requests at 12:00:59 and a further 3 requests at 12:01:00, this would allow the user to make 6 requests in a 2 second period. </li>
<li>It may also cause bursts of traffic across many clients. For example, if every client has used their quota for the pervious minute they may retry until they are allowed. This could cause many users to hit the server in the first second of the new window.</li>
</ul>
<h4>
Sliding Log</h4>
Sliding log rate limiting involves storing a history of requests for each user along with the associated time stamp. As new requests come in you count the number of requests for a period. Logs can be stored in a sorted set per user, where the key and value are the time stamp of the requests. Logs older than the allowed period are dropped.<br />
<br />
To recreate the previous example where the user is allowed 3 requests per 60 second window:<br />
<br />
<table class="relative-table wrapped confluenceTable tablesorter tablesorter-default stickyTableHeaders" role="grid" style="-webkit-text-stroke-width: 0px; background-color: white; border-collapse: collapse; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Noto Sans", Ubuntu, "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; margin: 0px; orphans: 2; overflow-x: auto; padding: 0px; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; width: 537px; word-spacing: 0px;"><colgroup><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col></colgroup><thead class="tableFloatingHeader">
<tr class="tablesorter-headerRow" role="row"><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="0" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Time</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="1" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Action</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="2" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Set Value</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="3" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Additional</div>
</th></tr>
</thead><tbody>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:05</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151200050000: 15151200050000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:15</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151200050000: 15151200050000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151200150000: 15151200150000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:01</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151200050000: 15151200050000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151200150000: 15151200150000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201010000: 15151201010000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
<br /></div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:10</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151200150000: 15151200150000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201010000: 15151201010000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201100000: 15151201100000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">User is still under threshold as the initial request (15151200000000) was more than 60 seconds ago</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:40</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201010000: 15151201010000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201100000: 15151201100000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201400000: 15151201400000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:50</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201010000: 15151201010000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201100000: 15151201100000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201400000: 15151201400000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201500000: 15151201500000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">User is rejected as they have had 4 request in the last 60 seconds</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:02:20</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201400000: 15151201400000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151201500000: 15151201500000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
15151202200000:15151202200000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">Request is allowed as user is below threshold.</td></tr>
</tbody></table>
<br />
<h4>
Pros:</h4>
<ul>
<li>Allows high precision on the limits</li>
<li>Avoids bursts of data at the start of a window</li>
</ul>
<h4>
Cons:</h4>
<ul>
<li>It requires a set item per request so has a large memory requirement.</li>
<li>Failed requests may still be added to the set. This could mean that even rate limited requests that perform no action could block a user.</li>
<li>May require multiple commands to perform the update and expire of old keys. </li>
<ul>
<li>This can be made atomic by using a redis MULTI command or lua script.</li>
</ul>
</ul>
<h3>
Sliding Window</h3>
In a sliding window rate limiter, each user is given limits to be used within a time frame. These are broken up into smaller buckets that allow us to control the way limits are available over a more evenly distributed window. As requests are used they are removed from the smaller windows and as those small windows expire the tokens are re-added. You can add more precision to your requests by having multiple windows that work in parallel to even out he flow of traffic. The data would be stored in a has per user to allow access to the individual buckets. The key for each bucket would be the time stamp at the start of the window.<br />
<br />
A simplified version allowing 3 requests per 60 seconds in 15 second buckets is:<br />
<br />
<table class="relative-table wrapped confluenceTable tablesorter tablesorter-default stickyTableHeaders" role="grid" style="-webkit-text-stroke-width: 0px; background-color: white; border-collapse: collapse; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Noto Sans", Ubuntu, "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; margin: 0px; orphans: 2; overflow-x: auto; padding: 0px; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; width: 531px; word-spacing: 0px;"><colgroup><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col></colgroup><thead class="tableFloatingHeader">
<tr class="tablesorter-headerRow" role="row"><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="0" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Time</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="1" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Action</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="2" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Set Value</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="3" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Additional</div>
</th></tr>
</thead><tbody>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:05</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120000: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:15</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120000: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120015: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:01</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120015: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120100: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Timestamp for 1515120000 is expired so is deleted</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:10</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120015: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120100: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:40</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120100: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120130: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">Timestamp for 1515120015 expires</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:50</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120100: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120130: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120145: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">User is rejected as they have had 4 request in the last 60 seconds</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:02:20</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120130: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120145: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
1515120215: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Timestamp for 1515120100 expires.</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
Request is allowed as user is below back threshold.</div>
</td></tr>
</tbody></table>
<br />
<h4>
Pros:</h4>
<ul>
<li>Allows varying degrees of precision depending on bucket time frames.</li>
<li>Lower memory constraints than sliding log.</li>
<li>Avoids bursts of data at the start of a new window.</li>
</ul>
<h4>
Cons:</h4>
<ul>
<li>The update is not atomic so would require a redis lua script to control key counting / expiry.</li>
</ul>
<h3>
Sliding Tail</h3>
Sliding tail rate limiting is an alternative implementation of the sliding window algorithm. It can be also be thought of as a fixed window rate limiting with history. In this case, we store a count of the current fixed window and the previous window. When a request comes in we calculate usage based on the count of the number of requests in the current window + a weighted count of the previous window. For example, if the current window is 25% through it's time, then we use a weighted count including 75% of the previous count + the current window count. This count is used to create the current number of tokens used by the user.<br />
<br />
The key for the information is the same as the fixed window example where we use <span style="font-family: "courier new" , "courier" , monospace;">{id}_{windowTimestamp}</span>. A server can do an atomic INCR for the current window time stamp and a GET for the previous window. To improve efficiency after the first request for a time stamp the server could cache the previous window data as this would no longer change.<br />
<br />
To redo the example of 3 requests per 60 seconds:<br />
<br />
<table class="relative-table wrapped confluenceTable tablesorter tablesorter-default stickyTableHeaders" role="grid" style="-webkit-text-stroke-width: 0px; background-color: white; border-collapse: collapse; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Noto Sans", Ubuntu, "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; margin: 0px; orphans: 2; overflow-x: auto; padding: 0px; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; width: 591px; word-spacing: 0px;"><colgroup><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col></colgroup><thead class="tableFloatingHeader">
<tr class="tablesorter-headerRow" role="row"><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="0" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Time</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="1" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Action</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="2" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Counters Value</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="3" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Additional</div>
</th></tr>
</thead><tbody>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:05</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120000: 0 → 1</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">No previous window so only using current window</td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:15</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user1_1515120000: 1 → 2</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:01</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1_1515120000: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
user1_1515120100: 0 → 1</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Weighted count is (2 * ((60-1)/60)) + 1 = 2.9</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:10</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1_1515120000: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
user1_1515120100: 1 → 2</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Weighted count is (2 * ((60-10)/60)) + 2 = 3.6</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:40</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1_1515120000: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
user1_1515120100: 2 → 3</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Weighted count is (2 * ((60-40)/60)) + 3 = 3.6</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:50</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1_1515120000: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
user1_1515120100: 3 → 4</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Weighted count is (2 * ((60-50)/60)) + 4 = 4.3</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
User is above threshold so rejected</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:02:20</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1_1515120100: 4</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
user1_1515120200: 0 → 1</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
Updated to a new window so user1_1515120000 is dropped and we move to using the weighted count from user1_1515120100</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
<br /></div>
<div style="margin: 10px 0px 0px; padding: 0px;">
Weighted count is (4 * ((60-20)/60)) + 1 = 3.6</div>
</td></tr>
</tbody></table>
<br />
In the above example you can see that the effect is the same as the fixed window for normal usage. However, if the user sent any more requests before 12:2:31, they would be rejected.<br />
<h4>
Pros:</h4>
<ul>
<li>Low memory usage</li>
<li>Increment is atomic, although there are 2 extra commands</li>
<ul>
<li>EXPIRE of the current key</li>
<li>GET of the previous key</li>
</ul>
<li>Prevents bursts of data at the start of a new window</li>
<li>Could be tuned to be more or less permissive based on weighting of the previous window</li>
<li>Could be tuned based on whether to round up or down</li>
</ul>
<h4>
Cons:</h4>
<ul>
<li>Only an approximation of the last window as it assumes there was a constant rate of traffic.</li>
<li>New clients could send bursts of traffic at their first window</li>
<li>If using atomic increment rejected requests could still add to the count for the window</li>
</ul>
<h3>
Leaky Bucket</h3>
The leaky bucket as a meter algorithm (related to token bucket) describes how an API can be limited to avoid burstiness. Each user has a count of tokens which is incremented as a request comes in. If the counter is above the threshold (the bucket size), then additional requests are discarded. To "empty" the bucket and give the user more tokens we decrement the counter by a set amount every period until it reaches zero.<br />
<br />
An implementation could keep a hash which includes, the token count, and the time stamp of the last time the bucket was emptied. As each request comes in, it performs 2 operations:<br />
<ol>
<li>Decrement the token based on a steady counter</li>
<li>Add a token</li>
</ol>
If the number is below the bucket size, the request is allowed.<br />
<br />
In the following example, we allow a bucket size of 3 and a refresh rate of 1 per 20 seconds:<br />
<br />
<table class="relative-table wrapped confluenceTable tablesorter tablesorter-default stickyTableHeaders" role="grid" style="-webkit-text-stroke-width: 0px; background-color: white; border-collapse: collapse; color: #333333; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Noto Sans", Ubuntu, "Droid Sans", "Helvetica Neue", sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; margin: 0px; orphans: 2; overflow-x: auto; padding: 0px; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; width: 531px; word-spacing: 0px;"><colgroup><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col><col style="width: 0px;"></col></colgroup><thead class="tableFloatingHeader">
<tr class="tablesorter-headerRow" role="row"><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="0" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Time</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="1" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Action</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="2" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Set Value</div>
</th><th class="confluenceTh tablesorter-header sortableHeader tablesorter-headerUnSorted" data-column="3" role="columnheader" scope="col" style="background: 100% center no-repeat rgb(240, 240, 240); border: 1px solid rgb(221, 221, 221); color: #333333; cursor: pointer; font-weight: bold; min-width: 8px; padding: 7px 15px 7px 10px; text-align: left; user-select: none; vertical-align: top;" tabindex="0"><div class="tablesorter-header-inner" style="color: #333333; margin: 0px; padding: 0px;">
Additional</div>
</th></tr>
</thead><tbody>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:05</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 1515120000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">Add a token to the bucket</td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:00:15</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 1515120000</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:01</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 1515120100</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
The previous ts was 1515120000 which means we should decrement the token count by up to 3.</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
Then add this token</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:10</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 1515120100</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:40</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 1515120140</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 10px 0px 0px; padding: 0px;">
The previous ts was 1515120100 which means we should decrement the token count by up to 2.</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
Then add this token</div>
</td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:01:50</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 2</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 1515120140</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><br /></td></tr>
<tr role="row"><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">12:02:20</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;">user makes request</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
user1: {</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
tokens: 1</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
ts: 15151200220</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
}</div>
</td><td class="confluenceTd" colspan="1" style="border: 1px solid rgb(221, 221, 221); min-width: 8px; padding: 7px 10px; text-align: left; vertical-align: top;"><div style="margin: 0px; padding: 0px;">
The previous ts was 1515120140 which means we should decrement the token count by up to 2.</div>
<div style="margin: 10px 0px 0px; padding: 0px;">
Then add this token</div>
</td></tr>
</tbody></table>
<br />
<h4>
Pros:</h4>
<ul>
<li>Memory efficient</li>
<li>Prevents bursts of traffic</li>
</ul>
<h4>
Cons:</h4>
<ul>
<li>Requires a redis lock or lua script to update. </li>
<li>Harder to tune requirements and implement. </li>
</ul>
<br />
<h2>
Implementation</h2>
For our purposes we decided to choose the sliding tail algorithm as it provided some protection against bursts of traffic while having low processing and memory usage. It is also easy to implement using standard redis commands that allow for atomic operations.<br />
<h3>
Additional Details</h3>
<h4>
HTTP Responses</h4>
When rejecting a user who is over the limit we choose a <span style="font-family: "courier new" , "courier" , monospace;">429 Too Many Requests</span> response.<br />
<br />
In addition to the 429 rejection, all responses include HTTP headers to inform the user what their limit is, and how many tokens are remaining. There appears to be no consensus on the HTTP header to use for this information. From looking at responses to other APIs we decided to go with:<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">X-Rate-Limit-Limit</span> - to describe the limit the user has for this request</li>
<li><span style="font-family: "courier new" , "courier" , monospace;">X-Rate-Limit-Remaining</span> - to describe the number of tokens left for the user.</li>
</ul>
<h4>
Tokens Used for each Request</h4>
As some operations are related but have different processing requirements, we decided to allow a configurable number of tokens per request type. This means that for certain endpoints we can allow the same bucket of tokens to rate limit the different requests. For example, a request to<span style="font-family: inherit;"> <span style="font-family: "courier new" , "courier" , monospace;">GET /user</span> would use 1 token from the <span style="font-family: "courier new" , "courier" , monospace;">{id}:user:{timestamp}</span> bucket, however a request to <span style="font-family: "courier new" , "courier" , monospace;">POST /user</span> would remove 2 tokens from the same bucket.</span><span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<br />
<br />
<br />twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-64261337453167832042017-10-05T23:46:00.002+01:002017-10-06T20:01:03.744+01:00Reverse engineering the EPH Controls Ember APII have recently had my home heating upgraded with a new gas boiler and heating controls. While looking at the controls my plumber recommended the thermostat and controller <a href="http://emberapp.ephcontrols.com/" target="_blank">EPH Controls</a>. I was leaning more towards a system like Hive or Honeywell but decided to give this one a go as it did tick most of the boxes I needed. It includes 2 zone system (heating and hot water) that can be controlled via the thermostat locally in the house or via an <a href="https://play.google.com/store/apps/details?id=com.ephcontrols.ember" target="_blank">app</a> remotely.<br />
<br />
The missing components to the EPH Ember system were integration into other home automation systems and voice control systems (e.g. Google Assistant). I decided it would be a good learning experience to reverse engineer the API and add it to <a href="https://home-assistant.io/" target="_blank">home assistant</a>, which I use for some other home automation controls. I have completed the first half of my task, which is reverse engineering the API, and from this have created a <a href="https://github.com/ttroy50/PyEphEmber" target="_blank">python library</a> which can be used to control / monitor the system. This blog post will explain the steps I used to get the API details from their app / server.<br />
<br />
The basic steps used to do this were: <br />
<ol>
<li>Figure out how the app and server communicate</li>
<li>Capture that traffic </li>
<li>Decode and examine the traffic</li>
</ol>
<h3>
Figure out how the app and server communicate </h3>
The very first thing to do is to find out how the app and server communicate. To do this I installed <a href="https://play.google.com/store/apps/details?id=jp.co.taosoftware.android.packetcapture" target="_blank">tPacketCapture</a> on my android phone and sniffed the network traffic on my phone. Once I had the packet capture on my phone, I transferred it to my laptop and opened it with wireshark. From this I could see a number of different connections going out from my phone. Once I eliminated the traffic going to google, I could see network traffic going from my phone to an IP 40.127.170.98 on port 443.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvp_sf8xhJ4lAezvBjchDZIjYLs85sYn46HdFkC4Xp7efVT1cPZ2nDvmrrLslpXGhGGuf8SfjW8pkT6wVAgi1JjI8oxTI99n9ezOPGA3wxesIgka4N5Qz_ya9tFnPulRfv-1Xu7N3tlru6/s1600/Screenshot+from+2017-10-05+23-05-04.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="559" data-original-width="937" height="190" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvp_sf8xhJ4lAezvBjchDZIjYLs85sYn46HdFkC4Xp7efVT1cPZ2nDvmrrLslpXGhGGuf8SfjW8pkT6wVAgi1JjI8oxTI99n9ezOPGA3wxesIgka4N5Qz_ya9tFnPulRfv-1Xu7N3tlru6/s320/Screenshot+from+2017-10-05+23-05-04.png" width="320" /></a></div>
<br />
As 443 is the port used by HTTPs, I could be fairly confident that that is the protocol used between server and client. HTTPs meant it was secure (good for my privacy) but (slightly) harder to decipher. However, as HTTPs is widely used there are plenty of tools available to sniff the connection when you are in control of the device and network.<br />
<h3>
Capture the traffic </h3>
The tool I chose for this was <a href="https://www.charlesproxy.com/" target="_blank">Charles Proxy</a> a HTTP(s) proxy which allows you to capture and display HTTP(s) traffic. Once installed you should configure it to do SSL Proxying, get the root certificate, and install it on your device. Some of these steps are explained <a href="https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/" target="_blank">here</a>. In later steps you will need the IP address of the computer running charles and the port that it is listening on (Proxy > Proxy Settings > Port)<br />
<br />
On your android phone, you need to edit the wifi network settings for the network that you are connected to and have an instance of charles running on to use a proxy and point it to your charles install. To do this on Android, follow these steps:<br />
<ul>
<li>Open your wifi settings.</li>
<li>Long click on the wifi network name and select modify network.</li>
<li>Select Advanced Options</li>
<li>Select Proxy > Manual</li>
<li>Enter the IP address of the server running charles</li>
<li>Enter the port charles is listening on (8888)</li>
<li>Save </li>
</ul>
Your phone should now route all HTTP(s) traffic through your charles instance. Once you launch the ember app you should see traffic going to it's server at https://ember.ephcontrols.com<br />
<h3>
Decode and examine the traffic</h3>
As mentioned earlier, this proves that the API is using HTTPS to communicate between the app and the server. Charles can show you all information about the HTTP traffic including URL, headers, data.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglRETP-TmNVI-OG-sIIozD-Hm-w4i1g69ErGhtCvTYfqaJnaBRYvA4-8_9lDk0Cj7KrVyHWNC6R-X_jSy9iHUcYvaGSwqH5F4UvIZmyKEYVVpTzm1iGw5KjjlNs9SjSfjhXkeGIS1vFYPP/s1600/Screenshot+from+2017-10-05+23-36-56.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="598" data-original-width="811" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglRETP-TmNVI-OG-sIIozD-Hm-w4i1g69ErGhtCvTYfqaJnaBRYvA4-8_9lDk0Cj7KrVyHWNC6R-X_jSy9iHUcYvaGSwqH5F4UvIZmyKEYVVpTzm1iGw5KjjlNs9SjSfjhXkeGIS1vFYPP/s320/Screenshot+from+2017-10-05+23-36-56.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Looking at the hierarchy of the folders in charles shows the URLs that are used. These include URLs such as<br />
<ul>
<li>api/Account/RefreshToken</li>
<li>api/Home/GetHomeById?homeId=1234</li>
</ul>
By looking at the contents of these requests I can see that the server and client communicate by sending JSON encoded messages between them. As JSON is a widely used and text based protocol it was then just a matter of reading the charles logs to decipher the full protocol.<br />
<br />
By running various scenarios in the app (e.g. login, check temperature, boost heating) I was able to capture most API calls and have documented them on <a href="https://github.com/ttroy50/pyephember/blob/master/API.md" target="_blank">github</a>.<br />
<br />
<br />
<br />twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com1tag:blogger.com,1999:blog-994665793725748583.post-22911359621033605812017-08-06T17:44:00.004+01:002017-08-06T17:48:00.130+01:00CUDA in DockerTo use <a href="http://www.nvidia.com/object/cuda_home_new.html" target="_blank">CUDA</a> within docker to take advantage of parallel programming on your GPU you need to expose your GPU inside the docker container. To do so you can use the <a href="https://github.com/NVIDIA/nvidia-docker" target="_blank">nvidia-docker</a> extension to expose your NVIDIA graphics card and drivers inside the container.<br />
<br />
<h3>
Requirements</h3>
<h4>
NVIDIA Driver </h4>
The first major requirement is to make sure you are using an NVIDIA graphics card and the NVIDIA propriety driver. On Ubuntu you can enable this from Software & Updates:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYjaP2uT0keIZQcpRAdS9KcSkTJmJNy7fWNQahIml2ASAN8XoBNXUQsvKhhltoPEpAEKlJDq0o_qDkOnqzqqhNaU-bb0Qev8T8-7u29OtFRD-BG51sjc7dOMhOK4z-6HMq9N8F3AMUhmHi/s1600/Screenshot+from+2017-08-06+16-11-47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="470" data-original-width="843" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYjaP2uT0keIZQcpRAdS9KcSkTJmJNy7fWNQahIml2ASAN8XoBNXUQsvKhhltoPEpAEKlJDq0o_qDkOnqzqqhNaU-bb0Qev8T8-7u29OtFRD-BG51sjc7dOMhOK4z-6HMq9N8F3AMUhmHi/s320/Screenshot+from+2017-08-06+16-11-47.png" width="320" /></a></div>
<br />
<br />
If using a laptop (or intel chip that includes integrated graphics) you may also need to make sure that you have selected the NVIDIA graphics card as the one in use. Once the driver is installed you can select the card using the NVIDIA X Server Settings applications:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7kXO0mfjc9gyJV4-TG5wbUnZRGEW9EY5CYZ-ULOdGoyF4nuw7Lkf6p8AWVWXkeS2zx9Y44sVAbxFwrfs2B9-iUckeNnWY0B7OrvD3sgvcBTW3CA8IZZuY2hxcVUTfcdnM50NAjbzpZhsW/s1600/Screenshot+from+2017-08-06+16-11-50.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="618" data-original-width="666" height="296" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7kXO0mfjc9gyJV4-TG5wbUnZRGEW9EY5CYZ-ULOdGoyF4nuw7Lkf6p8AWVWXkeS2zx9Y44sVAbxFwrfs2B9-iUckeNnWY0B7OrvD3sgvcBTW3CA8IZZuY2hxcVUTfcdnM50NAjbzpZhsW/s320/Screenshot+from+2017-08-06+16-11-50.png" width="320" /></a></div>
<br />
<br />
If you had to change either of the above, restart your computer for them to take effect.<br />
<br />
You can tell if your NVIDIA card is running by using the nvidia-smi command line tool:<br />
<br />
<pre class="brush: bash, shell">$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 660M Off | 0000:01:00.0 N/A | N/A |
| N/A 62C P0 N/A / N/A | 236MiB / 1999MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
$
</pre>
<h4>
Docker and NVIDIA Docker</h4>
Obviously to use docker you must first install it. Follow the instructions on <a href="http://docker.com/" target="_blank">their site</a> to download and install the latest version.<br />
<br />
After install docker, you should then install the <a href="https://github.com/NVIDIA/nvidia-docker" target="_blank">nvidia docker extension</a>. Installers and instructions are available on the linked github page.<br />
<br />
<h3>
Running nvidia-docker</h3>
Once everything is installed you should be able to use the GPU in your container<br />
<br />
<pre class="brush: bash, shell">$ nvidia-docker run -it --rm nvidia/cuda nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 660M Off | 0000:01:00.0 N/A | N/A |
| N/A 62C P0 N/A / N/A | 236MiB / 1999MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
$ </pre>
<pre class="brush: bash, shell"> </pre>
As you can see this is the same as the above output when running the tool outside of the container on my native host. <br />
<h4>
Choosing Type of OS</h4>
There are a number of <a href="https://hub.docker.com/r/nvidia/cuda/" target="_blank">pre-built images</a> available using different versions of CUDA and popular OS / Containers, including:<br />
<ul>
<li>Ubuntu 16.04</li>
<li>Ubuntu 14.04</li>
<li>CentOS 7</li>
<li>CentOS 8</li>
</ul>
It is also possible to view the dockerfiles used to create these images and extract the required values to re-create your own images. For example, <a href="https://gitlab.com/nvidia/cuda/blob/ubuntu16.04/8.0/runtime/Dockerfile" target="_blank">this is the dockerfile</a> for an Ubuntu 16.04 and CUDA 8<br />
<br />
<br />twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-41157042847357527322017-08-06T12:26:00.001+01:002019-02-19T14:39:32.755+00:00Clang on RHEL 7 via RPMIn a <a href="https://hopstorawpointers.blogspot.ie/2017/02/build-clang-llvm-tooling-on-rhel-7.html" target="_blank">previous post</a> I discussed how to build CLang and LLVM from source on CentOS 7. That approach works but problems with it are:<br />
<ol>
<li>It takes a long time to recompile for each user.</li>
<li>It results in large install.</li>
</ol>
As a result of these drawbacks, I wanted to build an RPM that could result in a minimal install. Some investigation showed that the latest RPM I could find for CentOS 7 was on <a href="https://fedoraproject.org/wiki/EPEL" target="_blank">EPEL</a> and was for CLang 3.4. However, fedora does have an RPM for CLang 3.9. Taking the SRPM for this I was able to modify it to make an RPM for CentOS 7.<br />
<h3>
Preparation</h3>
<h4>
CMake </h4>
To start with we need a version of CMake greater than v3.4.3. The easiest way to do this is via the epel repository:<br />
<br />
<pre class="brush: bash, shell">$ yum install epel-release
$ yum install cmake3
$ cmake3 --version
cmake3 version 3.6.3 </pre>
<h4>
Rpmbuild</h4>
You need to install rpmbuild:<br />
<br />
<pre class="brush: bash, shell">$ yum install rpmbuild</pre>
<h3>
Spec files</h3>
The spec files are provided below and as mentioned are a modified version of the spec files from the fedora LLVM / CLang v3.9.1 SRPM.7.<br />
<h4>
LLVM</h4>
<pre class="brush: bash, shell"># Components enabled if supported by target architecture:
%ifarch %ix86 x86_64
%bcond_without gold
%else
%bcond_with gold
%endif
Name: llvm
Version: 3.9.1
Release: 1%{?dist}
Summary: The Low Level Virtual Machine
License: NCSA
URL: http://llvm.org
Source0: http://llvm.org/releases/%{version}/%{name}-%{version}.src.tar.xz
Source100: llvm-config.h
BuildRequires: cmake3
BuildRequires: zlib-devel
BuildRequires: libffi-devel
BuildRequires: ncurses-devel
%if %{with gold}
BuildRequires: binutils-devel
%endif
BuildRequires: libstdc++-static
Requires: %{name}-libs%{?_isa} = %{version}-%{release}
%description
LLVM is a compiler infrastructure designed for compile-time, link-time,
runtime, and idle-time optimization of programs from arbitrary programming
languages. The compiler infrastructure includes mirror sets of programming
tools as well as libraries with equivalent functionality.
%package devel
Summary: Libraries and header files for LLVM
Requires: %{name}%{?_isa} = %{version}-%{release}
Requires(posttrans): %{_sbindir}/alternatives
Requires(posttrans): %{_sbindir}/alternatives
%description devel
This package contains library and header files needed to develop new native
programs that use the LLVM infrastructure.
%package doc
Summary: Documentation for LLVM
BuildArch: noarch
Requires: %{name} = %{version}-%{release}
%description doc
Documentation for the LLVM compiler infrastructure.
%package libs
Summary: LLVM shared libraries
%description libs
Shared libraries for the LLVM compiler infrastructure.
%package static
Summary: LLVM static libraries
%description static
Static libraries for the LLVM compiler infrastructure.
%prep
%autosetup -n %{name}-%{version}.src
%build
mkdir -p _build
cd _build
# force off shared libs as cmake macros turns it on.
%cmake3 .. \
-DBUILD_SHARED_LIBS:BOOL=OFF \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-Bsymbolic -static-libstdc++" \
%if 0%{?__isa_bits} == 64
-DLLVM_LIBDIR_SUFFIX=64 \
%else
-DLLVM_LIBDIR_SUFFIX= \
%endif
\
-DLLVM_TARGETS_TO_BUILD="X86;AMDGPU;PowerPC;NVPTX;SystemZ;AArch64;ARM;Mips;BPF" \
-DLLVM_ENABLE_LIBCXX:BOOL=OFF \
-DLLVM_ENABLE_ZLIB:BOOL=ON \
-DLLVM_ENABLE_FFI:BOOL=ON \
-DLLVM_ENABLE_RTTI:BOOL=ON \
%if %{with gold}
-DLLVM_BINUTILS_INCDIR=%{_includedir} \
%endif
\
-DLLVM_BUILD_RUNTIME:BOOL=ON \
\
-DLLVM_INCLUDE_TOOLS:BOOL=ON \
-DLLVM_BUILD_TOOLS:BOOL=ON \
\
-DLLVM_INCLUDE_TESTS:BOOL=ON \
-DLLVM_BUILD_TESTS:BOOL=ON \
\
-DLLVM_INCLUDE_EXAMPLES:BOOL=ON \
-DLLVM_BUILD_EXAMPLES:BOOL=OFF \
\
-DLLVM_INCLUDE_UTILS:BOOL=ON \
-DLLVM_INSTALL_UTILS:BOOL=OFF \
\
-DLLVM_INCLUDE_DOCS:BOOL=ON \
-DLLVM_BUILD_DOCS:BOOL=OFF \
-DLLVM_ENABLE_DOXYGEN:BOOL=OFF \
\
-DLLVM_BUILD_LLVM_DYLIB:BOOL=ON \
-DLLVM_DYLIB_EXPORT_ALL:BOOL=ON \
-DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
-DLLVM_BUILD_EXTERNAL_COMPILER_RT:BOOL=ON \
-DLLVM_INSTALL_TOOLCHAIN_ONLY:BOOL=OFF
make %{?_smp_mflags}
%install
cd _build
make install DESTDIR=%{buildroot}
# fix multi-lib
mv -v %{buildroot}%{_bindir}/llvm-config{,-%{__isa_bits}}
mv -v %{buildroot}%{_includedir}/llvm/Config/llvm-config{,-%{__isa_bits}}.h
install -m 0644 %{SOURCE100} %{buildroot}%{_includedir}/llvm/Config/llvm-config.h
%check
cd _build
make check-all || :
%post libs -p /sbin/ldconfig
%postun libs -p /sbin/ldconfig
%post devel
%{_sbindir}/update-alternatives --install %{_bindir}/llvm-config llvm-config %{_bindir}/llvm-config-%{__isa_bits} %{__isa_bits}
%postun devel
[ $1 -eq 0 ] && %{_sbindir}/update-alternatives --remove llvm-config %{_bindir}/llvm-config-%{__isa_bits}
%files
%{_bindir}/*
%exclude %{_bindir}/llvm-config-%{__isa_bits}
%files libs
%{_libdir}/BugpointPasses.so
%{_libdir}/LLVMHello.so
%if %{with gold}
%{_libdir}/LLVMgold.so
%endif
%{_libdir}/libLLVM-3.9*.so
%{_libdir}/libLTO.so
%files devel
%{_bindir}/llvm-config-%{__isa_bits}
%{_includedir}/llvm
%{_includedir}/llvm-c
%{_libdir}/libLLVM.so
%{_libdir}/cmake/llvm
%files static
%{_libdir}/*.a
%changelog
* Sun Aug 06 2017 Thom Troy <twmatrim hopstorawpointers.blogspot.ie=""> - 3.9.1-1
- First build - spec a modified version of fedora25 SRPM
</twmatrim></pre>
<h4>
Compiler-rt</h4>
<br />
<pre class="brush: bash, shell">%ifarch s390 s390x
# only limited set of libs available on s390(x) and the existing ones (stats, ubsan) don't provide debuginfo
%global debug_package %{nil}
%endif
Name: compiler-rt
Version: 3.9.1
Release: 1%{?dist}
Summary: LLVM "compiler-rt" runtime libraries
License: NCSA or MIT
URL: http://llvm.org
Source0: http://llvm.org/releases/%{version}/%{name}-%{version}.src.tar.xz
BuildRequires: cmake3
BuildRequires: python
BuildRequires: llvm-devel = %{version}
BuildRequires: llvm-static = %{version}
%description
The compiler-rt project is a part of the LLVM project. It provides
implementation of the low-level target-specific hooks required by
code generation, sanitizer runtimes and profiling library for code
instrumentation, and Blocks C language extension.
%prep
%setup -q -n %{name}-%{version}.src
%build
mkdir -p _build
cd _build
%cmake3 .. \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DLLVM_CONFIG_PATH:FILEPATH=%{_bindir}/llvm-config-%{__isa_bits} \
\
%if 0%{?__isa_bits} == 64
-DLLVM_LIBDIR_SUFFIX=64 \
%else
-DLLVM_LIBDIR_SUFFIX= \
%endif
-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF # could be on?
make %{?_smp_mflags}
%install
cd _build
make install DESTDIR=%{buildroot}
# move sanitizer lists to better place
mkdir -p %{buildroot}%{_libdir}/clang/%{version}
for file in asan_blacklist.txt msan_blacklist.txt dfsan_blacklist.txt cfi_blacklist.txt dfsan_abilist.txt; do
mv -v %{buildroot}%{_prefix}/${file} %{buildroot}%{_libdir}/clang/%{version}/ || :
done
# move sanitizer libs to better place
mkdir -p %{buildroot}%{_libdir}/clang/%{version}/lib
mv -v %{buildroot}%{_prefix}/lib/linux/libclang_rt* %{buildroot}%{_libdir}/clang/%{version}/lib
mkdir -p %{buildroot}%{_libdir}/clang/%{version}/lib/linux/
pushd %{buildroot}%{_libdir}/clang/%{version}/lib
for i in *.a *.syms *.so; do
ln -s ../$i linux/$i
done
%check
cd _build
#make check-all
%files
%{_includedir}/*
%{_libdir}/clang/%{version}
%changelog
* Fri Jul 14 2017 Thom Troy <twmatrim hopstorawpointers.blogspot.com=""> - 3.9.1-1
- First build - spec a modified version of fedora25 SRPM
</twmatrim></pre>
<br />
<h4>
Compiler-rt</h4>
<br />
<pre class="brush: bash, shell">Name: clang
Version: 3.9.1
Release: 1%{?dist}
Summary: A C language family front-end for LLVM
License: NCSA
URL: http://llvm.org
Source0: http://llvm.org/releases/%{version}/cfe-%{version}.src.tar.xz
Source100: clang-config.h
BuildRequires: cmake3
BuildRequires: llvm-devel = %{version}
BuildRequires: libxml2-devel
BuildRequires: llvm-static = %{version}
BuildRequires: perl-generators
BuildRequires: ncurses-devel
Requires: %{name}-libs%{?_isa} = %{version}-%{release}
# clang requires gcc, clang++ requires libstdc++-devel
# - https://bugzilla.redhat.com/show_bug.cgi?id=1021645
# - https://bugzilla.redhat.com/show_bug.cgi?id=1158594
Requires: libstdc++-devel
Requires: gcc-c++
%description
clang: noun
1. A loud, resonant, metallic sound.
2. The strident call of a crane or goose.
3. C-language family front-end toolkit.
The goal of the Clang project is to create a new C, C++, Objective C
and Objective C++ front-end for the LLVM compiler. Its tools are built
as libraries and designed to be loosely-coupled and extensible.
%package libs
Summary: Runtime library for clang
Requires: compiler-rt%{?_isa} >= %{version}
%description libs
Runtime library for clang.
%package devel
Summary: Development header files for clang.
Requires: %{name}%{?_isa} = %{version}-%{release}
%description devel
Development header files for clang.
%package analyzer
Summary: A source code analysis framework
License: NCSA and MIT
BuildArch: noarch
Requires: %{name} = %{version}-%{release}
# not picked up automatically since files are currently not installed in
# standard Python hierarchies yet
Requires: python
%description analyzer
The Clang Static Analyzer consists of both a source code analysis
framework and a standalone tool that finds bugs in C and Objective-C
programs. The standalone tool is invoked from the command-line, and is
intended to run in tandem with a build of a project or code base.
%prep
%setup -q -n cfe-%{version}.src
%build
mkdir -p _build
cd _build
%cmake3 .. \
-DLLVM_LINK_LLVM_DYLIB:BOOL=ON \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DLLVM_CONFIG:FILEPATH=/usr/bin/llvm-config-%{__isa_bits} \
\
-DCLANG_ENABLE_ARCMT:BOOL=ON \
-DCLANG_ENABLE_STATIC_ANALYZER:BOOL=ON \
-DCLANG_INCLUDE_DOCS:BOOL=ON \
-DCLANG_INCLUDE_TESTS:BOOL=ON \
-DCLANG_PLUGIN_SUPPORT:BOOL=ON \
-DENABLE_LINKER_BUILD_ID:BOOL=ON \
\
-DCLANG_BUILD_EXAMPLES:BOOL=OFF \
%if 0%{?__isa_bits} == 64
-DLLVM_LIBDIR_SUFFIX=64 \
%else
-DLLVM_LIBDIR_SUFFIX= \
%endif
-DLIB_SUFFIX=
make %{?_smp_mflags}
%install
cd _build
make install DESTDIR=%{buildroot}
# multilib fix
mv -v %{buildroot}%{_includedir}/clang/Config/config{,-%{__isa_bits}}.h
install -m 0644 %{SOURCE100} %{buildroot}%{_includedir}/clang/Config/config.h
# remove git integration
rm -vf %{buildroot}%{_bindir}/git-clang-format
# remove editor integrations (bbedit, sublime, emacs, vim)
rm -vf %{buildroot}%{_datadir}/clang/clang-format-bbedit.applescript
rm -vf %{buildroot}%{_datadir}/clang/clang-format-sublime.py*
rm -vf %{buildroot}%{_datadir}/clang/clang-format.el
rm -vf %{buildroot}%{_datadir}/clang/clang-format.py*
# remove diff reformatter
rm -vf %{buildroot}%{_datadir}/clang/clang-format-diff.py*
%check
# requires lit.py from LLVM utilities
#cd _build
#make check-all
%files
%{_libdir}/clang/
%{_bindir}/clang*
%{_bindir}/c-index-test
%files libs
%{_libdir}/*.so.*
%{_libdir}/*.so
%files devel
%{_includedir}/clang/
%{_includedir}/clang-c/
%{_libdir}/cmake/
%dir %{_datadir}/clang/
%files analyzer
%{_bindir}/scan-view
%{_bindir}/scan-build
%{_bindir}/scan-build
%{_libexecdir}/ccc-analyzer
%{_libexecdir}/c++-analyzer
%{_datadir}/scan-view/
%{_datadir}/scan-build/
%{_mandir}/man1/scan-build.1.*
%changelog
* Sun Aug 06 2017 Thom Troy <twmatrim hopstorawpointers.blogspot.ie=""> - 3.9.1-1
- First build - spec a modified version of fedora25 SRPM
</twmatrim></pre>
<br />
<h4>
Include What You Use</h4>
<br />
<pre class="brush: bash, shell">Name: iwyu
Version: 0.7
Release: 1%{?dist}
Summary: C/C++ source files #include analyzer based on clang
License: NCSA
Source0: https://github.com/include-what-you-use/include-what-you-use/archive/clang_3.9.tar.gz
BuildRequires: cmake3
BuildRequires: clang-devel >= 3.9
BuildRequires: llvm-devel
BuildRequires: llvm-static
BuildRequires: zlib-devel
# Scripts are Python 2
BuildRequires: python2-devel
BuildRequires: ncurses-devel
# Virtual provide the long name
Provides: include-what-you-use = %{version}-%{release}
Provides: include-what-you-use%{?_isa} = %{version}-%{release}
ExclusiveArch: %{ix86} x86_64
%description
"Include what you use" means this: for every symbol (type, function, variable,
or macro) that you use in foo.cc (or foo.cpp), either foo.cc or foo.h
should #include a .h file that exports the declaration of that symbol. The
include-what-you-use tool is a program that can be built with the clang
libraries in order to analyze #includes of source files to find
include-what-you-use violations, and suggest fixes for them.
%prep
%autosetup -n include-what-you-use-clang_3.9
%build
mkdir build
cd build
%cmake3 -DIWYU_LLVM_LIB_PATH=%{_libdir}/llvm -DIWYU_LLVM_INCLUDE_PATH=%{_includedir} ..
%make_build
%install
%make_install -C build
cd %{buildroot}%{_bindir}
ln -s include-what-you-use iwyu
ln -s fix_includes.py fix_includes
ln -s iwyu_tool.py iwyu_tool
%check
# Need to have the clang header's at the correct relative path (see https://github.com/include-what-you-use/include-what-you-use/issues/100 )
ln -s %{_libdir} %{_lib}
cd build
PATH=$PWD:$PATH
ln -s ../fix_includes.py
ln -s ../fix_includes_test.py
ln -s ../iwyu_test_util.py
ln -s ../run_iwyu_tests.py
ln -s ../tests
%{__python2} run_iwyu_tests.py
%{__python2} fix_includes_test.py
%files
%{_bindir}/include-what-you-use
%{_bindir}/iwyu
%{_bindir}/fix_includes
%{_bindir}/fix_includes.py
%{_bindir}/iwyu_tool
%{_bindir}/iwyu_tool.py
%dir %{_datadir}/include-what-you-use
%{_datadir}/include-what-you-use/*.imp
* Sun Aug 06 2017 Thom Troy <twmatrim hopstorawpointers.blogspot.ie=""> - 0.7
- Update to work on centos 7
</twmatrim></pre>
<h4>
Building the spec files</h4>
To build the spec files run:<br />
<br />
<pre class="brush: bash, shell">$ spectool -g -R SPECS/llvm.spec
$ rpmbuild -ba SPECS/llvm.spec
$ sudo yum install -y RPMS/x86_64/llvm-static-3.9.1-1.el7.centos.x86_64.rpm \
RPMS/x86_64/llvm-devel-3.9.1-1.el7.centos.x86_64.rpm \
RPMS/x86_64/llvm-libs-3.9.1-1.el7.centos.x86_64.rpm \
RPMS/x86_64/llvm-3.9.1-1.el7.centos.x86_64.rpm
$ spectool -g -R SPECS/compiler-rt.spec
$ rpmbuild -ba SPECS/compiler-rt.spec
$ sudo yum install -y RPMS/x86_64/compiler-rt-3.9.1-1.el7.centos.x86_64.rpm
$ spectool -g -R SPECS/clang.spec
$ rpmbuild -ba SPECS/clang.spec
$ sudo yum install -y RPMS/x86_64/clang-3.9.1-1.el7.centos.x86_64.rpm \
RPMS/x86_64/clang-libs-3.9.1-1.el7.centos.x86_64.rpm \
RPMS/x86_64/clang-devel-3.9.1-1.el7.centos.x86_64.rpm
$ spectool -g -R SPECS/iwyu.spec
$ rpmbuild -ba SPECS/iwyu.spec
</pre>
<br />
This will result in you having all RPMS and SRPMS needed to install and use CLang and LLVM v3.9.1twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-24700712471789814652017-02-25T10:56:00.002+00:002017-02-25T11:05:18.165+00:00It's the end of the line When working on an install script recently, I came across one of those bugs that make you realise just how pedantic computer programming can be. I had a file that contains a list of yum package names and a script that read the file and did some work on them.<br />
<br />
PackageList.txt<br />
<br />
<pre class="brush: bash, shell">redis
python
</pre>
<br />
InstallScript.sh<br />
<br />
<pre class="brush: bash, shell">while read PKG; do
yum install -y ${PKG}
done < /path/to/PackageList.txt
</pre>
<br />
This file had been working fine as part of our installer for a number of iterations. As part of a developing a new feature I added a new package to the list and saved the file. Thinking this was such a small change that it would just work, I committed it and pushed the changes. However when running the script our tester complained that the new package was missing.<br />
<br />
I sat down to debug the issue by checked that the package existed, that the script hadn't changed, and that I had the package name correct. As part of this debugging I resaved the file and it worked again.<br />
<br />
After scratching my head, getting a cup of tea and doing some searching, I discovered that the posix standard specifies that a newline character should be added to the end of files. My editor of choice for development is Sublime Text,<a href="https://forum.sublimetext.com/t/make-saving-newline-at-eof-the-installation-default/9842" target="_blank"> which by default</a> doesn't add the newline character to the end of the file.<br />
<br />
In order to turn it on you should edit your preferences to change the following preference to true.<br />
<br />
<pre class="brush: plain">// Set to true to ensure the last line of the file ends in a
// newline character when saving
"ensure_newline_at_eof_on_save": true
</pre>
<br />
You may also see the symptom of this issue when committing files to source control and at the end of a diff you will see.<br />
<br />
<pre class="brush: bash, shell">\ No newline at end of file
</pre>
<br />
<br />
<span style="color: #204a87; font-weight: bold;"> </span>twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-11438588483937095292017-02-10T21:29:00.001+00:002017-02-10T21:38:54.914+00:00Inflation ProblemsDespite 64-bit operating systems being the default for over 10 years, some of the code I use is still compiled with <span style="font-family: "courier new" , "courier" , monospace;">"-m32"</span> for 32-bit mode. The reasons for this are mostly lack of management will and developer time. As I got time between projects, I decided to update the code so that we can release in both 32-bit and 64-bit mode. <br />
<br />
Upgrading the code to be ready for 64-bit mode proved to be a slow task that had many chances for errors. I hope that by showing these errors and some common fixes it helps others to also update their code.<br />
<h3>
Common Errors</h3>
<h3>
int or unsigned int instead of size_t</h3>
<br />
On a 32-bit system this isn't really a problem as all 3 types use a 32-bit integer, so you won't get errors. However, it's not portable and on a 64-bit Linux system, size_t is a 64-bit (unsigned) integer. This can cause issues with comparisons and overflow. For example:<br />
<br />
<pre class="brush: cpp">string s = "some string";
unsigned int pos = s.find("st");
if( pos == string::npos) {
// code that can never be hit
}
</pre>
<br />
The above causes issues because <span style="font-family: "courier new" , "courier" , monospace;">string::npos</span> can never be equal to <span style="font-family: "courier new" , "courier" , monospace;">pos</span> as the data type of an <span style="font-family: "courier new" , "courier" , monospace;">unsigned int</span> is too small to match <span style="font-family: "courier new" , "courier" , monospace;">string::npos</span>. <br />
<br />
<br />
This issue can be caught with the compiler flag <span style="font-family: "courier new" , "courier" , monospace;">-Wtype-limits</span>. Or preferably use <span style="font-family: "courier new" , "courier" , monospace;">-Werror=type=limits</span> to cause the compilation to fail with the following error<br />
<br />
<pre class="brush: bash, shell">error: comparison is always false due to limited range of data type [-Werror=type-limits]</pre>
<br />
As mentioned this can also cause overflow issues, for example:<br />
<br />
<pre class="brush: cpp">unsigned int pos = string::npos;
</pre>
<br />
This causes an overflow because <span style="font-family: "courier new" , "courier" , monospace;">string::npos </span>is too big to fit in a 32-bit integer.<br />
<br />
Again this can be caught by a compiler flag, in this case <span style="font-family: "courier new" , "courier" , monospace;">-Woverflow</span>. And again I recommend to use <span style="font-family: "courier new" , "courier" , monospace;">-Werror=overflow</span> to cause a compilation error.<br />
<h4>
Wrong printf arguments</h4>
<br />
The logger in our codebase uses printf style formatting for formatting log lines. As a result of this the most common warning on our 64-bit compile was related to this.<br />
<br />
The most common cause was related to the above assumption that a <span style="font-family: "courier new" , "courier" , monospace;">size_t</span> is a 32-bit integer. Below is an example of the warning showing this<br />
<br />
<pre class="brush: bash, shell">warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'size_t {aka long unsigned int}' [-Wformat=]
TRACE(("Insert at position [%u]", pos));</pre>
<br />
The fix that I used for this warning to use the <span style="font-family: "Courier New",Courier,monospace;">%zu</span> format specifier for <span style="font-family: "Courier New",Courier,monospace;">size_t</span>. This was introduced in the C99 standard and should be available in gcc and clang. However, it may not be available in some older versions of the Visual Studio compiler.<br />
<br />
<br />
<pre class="brush: cpp">TRACE(("Insert at position [%zu]", pos));</pre>
<br />
I have also seen the above error in relation to other types, for example <span style="font-family: "Courier New",Courier,monospace;">time_t</span>, <span style="font-family: "Courier New",Courier,monospace;">uintptr_t</span>, and <span style="font-family: "Courier New",Courier,monospace;">long</span>. If you are unsure of what the printf argument for a type is, then you can use helpful macros from the C <span style="font-family: "Courier New",Courier,monospace;">"inttypes.h"</span> header (<span style="font-family: "Courier New",Courier,monospace;"><cinttypes></span> if using C++11 or later). This includes macros with the printf arguments for various system typedefs. <br />
<br />
Note: Before C++11 you must define <span style="font-family: "Courier New",Courier,monospace;">__STDC_FORMAT_MACROS</span> before including this header. For example, to print a <span style="font-family: "Courier New",Courier,monospace;">uintptr_t</span> you can use the macro <span style="font-family: "Courier New",Courier,monospace;">PRIuPTR</span><br />
<br />
<br />
<pre class="brush: cpp">#define __STDC_FORMAT_MACROS 1
#include <inttypes.h>
bool MyList::insert(uintptr_t value)
{
....
TRACE(("value [%" PRIuPTR "]", value));</pre>
<h4>
Assuming the size of a type is always the same</h4>
<br />
Again this is somewhat related to the previous points. I saw a number of errors where it was assumed that a particular type was always the same length on different platforms.<br />
<br />
The 2 most common were pointers and long.<br />
<br />
In our code pointer length issues often manifest as the <span style="font-family: "Courier New",Courier,monospace;">printf</span> argument error, e.g. using <span style="font-family: "Courier New",Courier,monospace;">%08x</span> instead of <span style="font-family: "Courier New",Courier,monospace;">%p</span> but I also saw some cases where a pointer was cast to an int to pass it through a particular function. This would then cause it to then precision on a 64-bit system.<br />
<br />
<br />
In the case of long it appears that in many cases it was assumed that long was always a 32-bit integer. I came across a number of errors caused by using bitwise operations which assumed that a long was 32-bits. For example:<br />
<br />
<pre class="brush: cpp">long offset = getSomeValue();
if ( offset & (1 << 31) )</pre>
<br />
This causes errors because long is not guaranteed to be a 32-bit integer. If you need to guarantee a size then you should use the correct typedef for that sized integer from the C <span style="font-family: "Courier New",Courier,monospace;">"stdint.h"</span> header (<span style="font-family: "Courier New",Courier,monospace;"><cstdint></span> for C++11). e.g. <br />
<br />
<pre class="brush: cpp">#include <stdint.h>
int32_t i32 = get32bitInt();
int64_t i64 = get64bitint();
...</pre>
<br />
These can then be used in conjunction with the PRIxxx macros from inttypes.h if you need to log / format them<br />
<br />
Even with stdint.h there were some ambiguous types that were being cast to / from different types. An example of this was <span style="font-family: "Courier New",Courier,monospace;">time_t</span> which is not actually defined in a standard. After some googling and testing, I discovered it aligns to the same size as a long (4 bytes on a 32-bit arch, 8 bytes on 64-bit). So when we needed to pass a <span style="font-family: "Courier New",Courier,monospace;">time_t</span> value and can't use the <span style="font-family: "Courier New",Courier,monospace;">time_t </span>typedef I defaulted to using a long.<br />
<br />
<br />
At the end of the article I show a very simple test program and it's output on RedHat Linux. This shows how the size of types can change depending on compilation mode.<br />
<h4>
Using the wrong type with malloc</h4>
This issue is not actually related to the 64-bit port but the symptoms of it only manifested when we ran the code in 64-bit mode. <br />
<br />
There were a couple of blocks of code that were using <span style="font-family: "Courier New",Courier,monospace;">malloc</span> to get a block of code for an array and these were using the wrong type for the <span style="font-family: "Courier New",Courier,monospace;">sizeof</span> argument. For example, some code for a hash table included:<br />
<br />
<br />
<pre class="brush: cpp">typedef struct HT
{
int num_entries;
int size;
HTEntry **table;
} HT;</pre>
<br />
Then to initialize the table<br />
<br />
<pre class="brush: cpp">HT *newtable = NULL;
newtable = (HT*)malloc(sizeof(HT));
newtable->size = size;
newtable->table = (HTEntry**)malloc(sizeof(int)*size);</pre>
<br />
<br />
This has been deployed and run error free for a number of years in our 32-bit software release. However, as the sizeof an int and the size of pointers differ on 64-bit systems, it caused errors there.<br />
<br />
The correct code is:<br />
<br />
<pre class="brush: cpp">newtable->table = (HTEntry**)malloc(sizeof(HTEntry*)*size);</pre>
<br />
Unfortunately I was unable to catch this with any compiler warnings and it caused a crash when run. I had also run some static analyzers over the code which missed this.<br />
<h3>
Conclusions</h3>
The task of updating your code to make it 64-bit compatible is slow, however, can be made easier if you take care to listen to your tools. This includes enabling compiler warnings, making some warnings errors, and using static analysis tools. These will help catch many of the common errors that can occour.<br />
<br />
<br />
As for the benefit of updating, it will be worth it because:<br />
<ul>
<li> It will help improve compatibility. As most OSes and software projects are now released in 64-bit mode by default, there is less chance of finding an incompatible package</li>
<li>Allow access to new CPU instructions. Compiling with 64bit mode allows access to new instructions and registers. Some initial tests have shown that certain sections of code can be up to 10% faster.</li>
<li>Improved code. Keeping the code compiling and working in both environments may lead to more careful programming.</li>
</ul>
<h3>
References</h3>
<a href="http://www.drdobbs.com/cpp/porting-to-64-bit-platforms/226600156?pgno=1">http://www.drdobbs.com/cpp/porting-to-64-bit-platforms/226600156?pgno=1</a><br />
<br />
<a href="http://www.viva64.com/en/a/0004/">http://www.viva64.com/en/a/0004/</a><br />
<br />
<a href="http://www.drdobbs.com/parallel/multiplatform-porting-to-64-bits/184406427">http://www.drdobbs.com/parallel/multiplatform-porting-to-64-bits/184406427</a><br />
<br />
<h4>
Test program to check common sizes</h4>
In order to check sizes, I created a simple test program that will print out the sizes for some common types:<br />
<br />
<pre class="brush: cpp">#include <iostream>
#include <cstdio>
#include <stdint.h>
#include <inttypes.h>
using namespace std;
int main()
{
cout << "sizeof(int) : " << sizeof(int) << std::endl;
cout << "sizeof(unsigned long) : " << sizeof(unsigned long) << std::endl;
cout << "sizeof(long int) : " << sizeof(long int) << std::endl;
cout << "sizeof(long long int) : " << sizeof(long long int) << std::endl;
cout << "sizeof(int32_t) : " << sizeof(int32_t) << std::endl;
cout << "sizeof(int64_t) : " << sizeof(int64_t) << std::endl;
cout << "sizeof(double) : " << sizeof(double) << std::endl;
cout << "sizeof(float) : " << sizeof(float) << std::endl;
cout << "sizeof(size_t) : " << sizeof(size_t) << std::endl;
cout << "sizeof(intptr_t) : " << sizeof(intptr_t) << std::endl;
cout << "sizeof(uintptr_t) : " << sizeof(uintptr_t) << std::endl;
cout << "sizeof(void*) : " << sizeof(void*) << std::endl;
cout << "sizeof(char) : " << sizeof(char) << std::endl;
}
</pre>
<br />
To compile and run, you can use:<br />
<br />
<pre class="brush: bash, shell">$> .g++ sizes.cpp -m32 -o t32.sizes
$> ./t32.sizes
sizeof(int) : 4
sizeof(unsigned long) : 4
sizeof(long int) : 4
sizeof(long long int) : 8
sizeof(int32_t) : 4
sizeof(int64_t) : 8
sizeof(double) : 8
sizeof(float) : 4
sizeof(size_t) : 4
sizeof(intptr_t) : 4
sizeof(uintptr_t) : 4
sizeof(void*) : 4
sizeof(char) : 1
$> .g++ sizes.cpp -o t64.sizes
$> ./t64.sizes
sizeof(int) : 4
sizeof(unsigned long) :8
sizeof(long int) : 8
sizeof(long long int) : 8
sizeof(int32_t) : 4
sizeof(int64_t) : 8
sizeof(double) : 8
sizeof(float) : 4
sizeof(size_t) : 8
sizeof(intptr_t) : 8
sizeof(uintptr_t) : 8
sizeof(void*) : 8
sizeof(char) : 1
</pre>
<br />
<br />
As you can see there are a number of types that have different sizes. These will be the same on all Linux systems, however they aren't guaranteed across different operating systems. twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-47566316031070717322017-02-02T22:41:00.000+00:002017-02-02T22:41:08.893+00:00Build Clang & LLVM tooling on RHEL 7<a href="https://clang.llvm.org/">Clang</a> is a C (and C++) front-end for the <a href="http://www.llvm.org/" target="_blank">LLVM</a> compiler. It provides a fast compiler with really good error messages and great support for writing code analysis and formatting tools. Some of the official tools include:<br />
<ul>
<li><a href="https://clang-analyzer.llvm.org/" target="_blank">Clang Static Analyzer</a></li>
<li><a href="https://clang.llvm.org/docs/ClangFormat.html" target="_blank">Clang Format</a></li>
<li><a href="http://clang.llvm.org/extra/clang-tidy/" target="_blank">Clang Tidy</a></li>
<li><a href="http://clang.llvm.org/extra/clang-rename.html" target="_blank">Clang Rename</a></li>
</ul>
Third party tools built on top of the clang tooling (and libclang libraries) include: <br />
<ul>
<li><a href="https://include-what-you-use.org/" target="_blank">Include What You Use</a></li>
<li><a href="https://valloric.github.io/YouCompleteMe/" target="_blank">YouCompleteMe</a></li>
</ul>
A good talk by Chandler Carruth about some of the above tools and the future direction for Clang tooling is <a href="https://www.youtube.com/shared?ci=Ve7K6ginDyY" target="_blank">available on YouTube</a><br />
<h3>
Installing Clang </h3>
<h4>
Redhat 7</h4>
On RedHat 7, Clang is not included in the official repositories, however older versions (v3.4) are included in the <a href="https://dl.fedoraproject.org/pub/epel/7/x86_64/c/" target="_blank">epel</a> repository.<br />
<br />
If you are unable to use the epel repository, or want a newer version of clang, the below script can be used to get and install v3.9.1 of llvm, clang, clang tools and the include what you use tool.<br />
<br />
<pre class="brush: bash, shell">mkdir clang_llvm_391_build
cd clang_llvm_391_build
svn co http://llvm.org/svn/llvm-project/llvm/tags/RELEASE_391/final llvm
cd llvm/tools
svn co http://llvm.org/svn/llvm-project/cfe/tags/RELEASE_391/final clang
cd ../..
cd llvm/tools/clang/tools
svn co http://llvm.org/svn/llvm-project/clang-tools-extra/tags/RELEASE_391/final extra
cd ../../../..
cd llvm/projects
svn co http://llvm.org/svn/llvm-project/compiler-rt/tags/RELEASE_391/final compiler-rt
cd ../..
#cd llvm/projects
#svn co http://llvm.org/svn/llvm-project/libcxx/tags/RELEASE_391/final libcxx
#cd ../..
cd llvm/tools/clang/tools
git clone https://github.com/include-what-you-use/include-what-you-use.git
cd include-what-you-use
git checkout clang_3.9
cd ..
echo "" >> CMakeLists.txt
echo "add_subdirectory(include-what-you-use)" >> CMakeLists.txt
cd ../../../..
mkdir llvm.build
cd llvm.build
cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/opt/software/clang -DCMAKE_BUILD_TYPE=Release ../llvm
ninja
mkdir -p /opt/software/clang
cmake -DCMAKE_INSTALL_PREFIX=/opt/software/clang -P cmake_install.cmake
</pre>
<br />
<br />
As you can see this installs the software to <span style="font-family: "courier new" , "courier" , monospace;">/opt/software/clang </span>If you want to install to a different location change the <span style="font-family: "courier new" , "courier" , monospace;">CMAKE_INSTALL_PREFIX</span> locations in the line 26 and 29.<br />
<br />
The script doesn't compile the version of the C++ standard library (libcxx) available with Clang as I had compiler errors when building it with the default version of gcc (v4.8.5) available with RHEL 7.3<br />
<h4>
Redhat 6</h4>
For RHEL 6, there is also a <a href="https://dl.fedoraproject.org/pub/epel/6/x86_64/" target="_blank">epel</a> repository with v3.4 available. However, if you want a later version of Clang you have some hoops to run through.<br />
<br />
This is because Clang requires a C++11 compiler and Clang v3.9.1, mentioned above, requires at least v4.8 of gcc. The version of gcc available on RHEL 6 is too old and you have to manually build a later version before you can build Clang. You can find instructions on doing so from <a href="http://btorpey.github.io/blog/2015/01/02/building-clang/" target="_blank">this blog post</a>.<br />
<h3>
Using Clang</h3>
<h4>
Compiling </h4>
To build your software using Clang with CMake you should override the <span style="font-family: "courier new" , "courier" , monospace;">CMAKE_C_COMPILER</span> and <span style="font-family: "courier new" , "courier" , monospace;">CMAKE_CXX_COMPILER</span> variables. Using the install from my script above this would be done using<br />
<br />
<pre class="brush: bash, shell">$ cmake -DCMAKE_C_COMPILER=/opt/software/clang/bin/clang -DCMAKE_CXX_COMPILER=/opt/software/clang/bin/clang++ ..
$ make
</pre>
<br />
You can see more details in my <a href="https://github.com/ttroy50/cmake-examples/tree/master/01-basic/I-compiling-with-clang" target="_blank">cmake-examples</a> GitHub repository.<br />
<br />
Similar methods of overriding the C and C++ compiler environments may work with other build tools. e.g. using <span style="font-family: "courier new" , "courier" , monospace;">CC</span> and <span style="font-family: "courier new" , "courier" , monospace;">CXX</span> with Makefiles.<br />
<h4>
Using Clang Static Analyzer</h4>
Using the Clang Static Analyzer is easy too as it includes a tool <span style="font-family: "courier new" , "courier" , monospace;">scan-build</span> which can be used to scan your source code at the same time as it builds it<br />
<br />
<pre class="brush: bash, shell">$ /opt/software/clang/bin/scan-build cmake ..
$ /opt/software/clang/bin/scan-build make
</pre>
<br />
On RedHat the above will use gcc to build your software while scanning it with the Clang Static Analyzer.<br />
<br />
To get extra coverage for your code I also recommend to use clang to compile it. This can be done at the same time as your static analysis by using the <span style="font-family: "courier new" , "courier" , monospace;">--use-cc</span> and <span style="font-family: "courier new" , "courier" , monospace;">--use-c++</span> flags for scan-build<br />
<br />
<pre class="brush: bash, shell">$ /opt/software/clang/bin/scan-build --use-cc=/opt/software/clang/bin//clang --use-c++=/opt/software/clang/bin//clang++ cmake ..
$ /opt/software/clang/bin/scan-build --use-cc=/opt/software/clang/bin//clang --use-c++=/opt/software/clang/bin//clang++ make
</pre>
<br />
<h3>
Advantages of having Clang Available</h3>
The main reason I have for using Clang on RedHat is to get access to it's tooling and static analyzer.<br />
<br />
However as a side effect of this it also makes the compiler available for use. Using this second compiler can give you more chance of finding errors. For example, when compiling with Clang I had an error:<br />
<br />
<pre class="brush: plain"> file included from /path/to/myclass.cpp:22:
/path/to/logger.h:1:9: warning: '_LIBMYLIB_LOGGER_H_' is used as a header guard here, followed by
#define of a different macro [-Wheader-guard]
#ifndef _LIBMYLIB_LOGGER_H_
^~~~~~~~~~~~~~~~~
/path/to/logger.h:2:9: note: '_LINMYLIB_LOGGER_H_' is defined here; did you mean '_LIBMYLIB_LOGGER_H_'?
#define _LINMYLIB_LOGGER_H_
^~~~~~~~~~~~~~~~~
_LIBMYLIB_LOGGER_H_
6 warnings generated.
</pre>
<br />
This did not cause any errors or warnings on my version of GCC and while it didn't cause any issues (because I only included that header once), it could potentially have lead to a later error. twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-22740529732034899832017-01-29T17:37:00.001+00:002017-01-29T17:51:29.339+00:00Uploading a VirtualBox VM to an Amazon EC2 AMIThere are many blog posts about uploading a VirtualBox VM image to an AWS AMI. However, many are out of date or only cover part of the process. Below I try to describe the current easiest method to convert your custom VM to an AMI.<br />
<h2>
Reasons for wanting a custom AMI</h2>
As part of improving test coverage I am looking into using Amazon EC2 to launch snapshots of test slaves. These snapshots will be partially configured (e.g. basic postgres, redis, and nginx configuration). It would be possible to use a standard Amazon supplied AMI and user data to perform this. However, as I want to have these VMs be as close as possible to our customer installed images, I would prefer to create a local image from our custom install media and then upload it to Amazon.<br />
<h2>
Initial Investigation</h2>
The first step to achieving this is to search on Google. This obviously turned up the AWS documentation which shows that it is possible to upload your own AMIs. However, it seems to mostly focus on VMware and HyperV. For my purposes, I would much prefer to use a simple VirtualBox image.<br />
<br />
After, that I narrowed the search and found a number of blog posts describing the procedure. Many of these describe a convoluted process of creating an image, converting an image to RAW format and then uploading that.<br />
<br />
As the AWS documentation describes it being possible to use VMDK and VHD images, and these are supported by VirtualBox, I decided to dig deeper and found that it was possible to upload these if they are created by VirtualBox.<br />
<h2>
Steps to achieve it</h2>
<h3>
Prepare your VM</h3>
Create your VM in VirtualBox using the standard methods. The only specific requirement is that your hard disk file type should be VMDK (Virtual Machine Disk).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1EF_gcvsqljtHDEMYkWtvZPS53FOSWpV2TkwB_WU3jmq7Rt-n13cq1icCYzFkdufCjpRAJNsPDvCQfue7nzNeA1YohVO2Q4HOn7KDinIITFcflrPSBtQyDURWrbn8HLEAQ4panju8oh4N/s1600/vmdk_selection.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1EF_gcvsqljtHDEMYkWtvZPS53FOSWpV2TkwB_WU3jmq7Rt-n13cq1icCYzFkdufCjpRAJNsPDvCQfue7nzNeA1YohVO2Q4HOn7KDinIITFcflrPSBtQyDURWrbn8HLEAQ4panju8oh4N/s320/vmdk_selection.png" width="320" /></a></div>
<br />
<br />
In this example, I used a dynamically allocated hard disk file of size 16GB. It should be possible to use a fixed hard disk but that would take longer to upload. For a comparison of how much space you can save, in my simple example, the 16GB dynamically allocated VMDK will be exported as a 570MB OVA file.<br />
<h4>
Install your VM</h4>
The next step is to install the operating system on your OS and configure any required packages. For Linux, this will include an SSH server and network configured as DHCP. For more details, the Amazon documentation describes the main <a href="http://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html" target="_blank">prerequisites</a> and configuration <a href="http://docs.aws.amazon.com/vm-import/latest/userguide/prepare-vm-image.html" target="_blank">requirements</a> for VM.<br />
<br />
Note: Some posts say you have to installing cloud-init onto the OS. This is recommended but I found that it wasn't a hard requiremen<span id="goog_1550265342"></span><span id="goog_1550265343"></span>t for my needs.<br />
<h4>
Export your VM</h4>
Not that we have our VM ready for use, we have to export it. In VirtualBox, select File > Export Appliance. And export your virtual machine as below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJQ7nVp9OJ-AdJFIs0gvP1t7pb_WWcGTy6c-K13KC9_KizofBDxCpLNzb75qB37E64TSWaRvs3lmN7pExq8ZngUOylG9rOTnFs2GuZuLGxfX2z6ivlndQJHCVeUWUMDbRbr9ugdJhTQpO6/s1600/export.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="208" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJQ7nVp9OJ-AdJFIs0gvP1t7pb_WWcGTy6c-K13KC9_KizofBDxCpLNzb75qB37E64TSWaRvs3lmN7pExq8ZngUOylG9rOTnFs2GuZuLGxfX2z6ivlndQJHCVeUWUMDbRbr9ugdJhTQpO6/s320/export.png" width="320" /></a></div>
<br />
<br />
You now have a virtual machine in OVA format which is ready for upload to Amazon for use as an AMI. <br />
<h3>
Upload to Amazon</h3>
The steps to upload to Amazon are described in the AWS <a href="http://docs.aws.amazon.com/vm-import/latest/userguide/import-vm-image.html" target="_blank">documentation</a>, but I will repeat the basic steps here for completeness of this post.<br />
<h4>
Account Permissions </h4>
As described <a href="http://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#iam-permissions-image" target="_blank">here</a> the your AIM user should have the following permissions<br />
<br />
<pre class="brush: plain">{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:DeleteBucket",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": ["arn:aws:s3:::mys3bucket","arn:aws:s3:::mys3bucket/*"]
},
{
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:PutRolePolicy"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CancelConversionTask",
"ec2:CancelExportTask",
"ec2:CreateImage",
"ec2:CreateInstanceExportTask",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DescribeConversionTasks",
"ec2:DescribeExportTasks",
"ec2:DescribeInstanceAttribute",
"ec2:DescribeInstanceStatus",
"ec2:DescribeInstances",
"ec2:DescribeTags",
"ec2:ImportInstance",
"ec2:ImportVolume",
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:ImportImage",
"ec2:ImportSnapshot",
"ec2:DescribeImportImageTasks",
"ec2:DescribeImportSnapshotTasks",
"ec2:CancelImportTask"
],
"Resource": "*"
}
]
}</pre>
<h4>
Upload your image to S3</h4>
Create an S3 bucket (in the same region that you will want to run your EC2 instances in). Then upload the previously exported OVA file to that bucket.<br />
<h4>
Install the AWS CLI</h4>
Install the AWS CLI tools as described <a href="http://docs.aws.amazon.com/cli/latest/userguide/installing.html" target="_blank">here</a>. I used a python virtual environment to keep the tools separate from my standard Ubuntu install.<br />
<br />
My steps were:<br />
<br />
<pre class="brush: bash, shell">$ sudo pip install venv
$ mkdir aws
$ cd aws
$ virtualenv awsvenv
$ source awsvenv/bin/activate
(awsvenv)$ pip install awscli
</pre>
<br />
You now have the AWS CLI installed and you need to configure it to work with your AMI user. You can do this by running<br />
<br />
<pre class="brush: bash, shell">(awsvenv)$ aws configure
AWS Access Key ID: [your access id key]
AWS Secret Access Key: [your access key secret]
Default region name: [your region id e.g. eu-west-1]
Default output format: [text or json]
</pre>
<h4>
Create a VM Import Service Role</h4>
You have to create a role to allow you to import your VM and download images from S3. Create a file trust-policy.json<br />
<br />
<pre class="brush: plain">{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "vmie.amazonaws.com" },
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals":{
"sts:Externalid": "vmimport"
}
}
}
]
}
</pre>
<br />
Using the AWS CLI create the role vmimport using this file.<br />
<br />
<pre class="brush: bash, shell"> aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json </pre>
<br />
Create a file role-policy.json with the following policy, where <span style="font-family: "courier new" , "courier" , monospace;">disk-image-file-bucket</span> is the bucket where the disk images are stored:<br />
<br />
<pre class="brush: plain">{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::disk-image-file-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::disk-image-file-bucket/*"
]
},
{
"Effect": "Allow",
"Action":[
"ec2:ModifySnapshotAttribute",
"ec2:CopySnapshot",
"ec2:RegisterImage",
"ec2:Describe*"
],
"Resource": "*"
}
]
}
</pre>
<br />
Attach the policy to the role created above.<br />
<br />
<pre class="brush: bash, shell">aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json
</pre>
<h4>
Import the VM</h4>
You are now ready to import the OVA file you uploaded to S3 as an AMI. You can accomplish this using the AWS CLIs <span style="font-family: "courier new" , "courier" , monospace;">import-image</span> command. Create a file containers.json describing the S3 bucket and file.<br />
<br />
<pre class="brush: plain">[
{
"Description": "RHEL 7.3 Blog OVA",
"Format": "ova",
"UserBucket": {
"S3Bucket": "my-import-bucket",
"S3Key": "ami-blogpostvm.ova"
}
}]
</pre>
<br />
<pre class="brush: bash, shell">(awsvenv)$ aws ec2 import-image --description "RHEL 7.3 Blog OVA" --disk-containers file://containers.json
RHEL 7.3 Blog OVA import-ami-fg123456 2 active pending
SNAPSHOTDETAILS 0.0 OVA
USERBUCKET my-import-bucket ami-blogpostvm.ova
</pre>
<br />
Your image is now being imported. This can take a while depending on the size of the image. You can check on the status of the image using the <span style="font-family: "courier new" , "courier" , monospace;">describe-import-image-tasks</span> command and the task id returned from the <span style="font-family: "courier new" , "courier" , monospace;">import-image</span> command<br />
<br />
<pre class="brush: bash, shell">(awsvenv)$ aws ec2 describe-import-image-tasks --import-task-ids import-ami-fg123456
IMPORTIMAGETASKS RHEL 7.3 Blog OVA import-ami-fg123456 2 active pending
SNAPSHOTDETAILS 0.0 OVA
USERBUCKET my-import-bucket ami-blogpostvm.ova
</pre>
<br />
You should see the status of the image going through the following stages<br />
<ul>
<li>pending</li>
<li>converting</li>
<li>updating</li>
<li>active booting</li>
<li>active booted</li>
<li>active preparing ami</li>
<li>complete</li>
</ul>
<br />
<pre class="brush: bash, shell">(awsvenv)$ aws --output json ec2 describe-import-image-tasks --import-task-ids import-ami-fg123456
{
"ImportImageTasks": [
{
"Description": "RHEL 7.3 Blog OVA",
"LicenseType": "BYOL",
"ImageId": "ami-aa123pp",
"ImportTaskId": "import-ami-fg123456",
"Status": "completed",
"Architecture": "x86_64",
"SnapshotDetails": [
{
"SnapshotId": "snap-0845ad45ad45ad45",
"DeviceName": "/dev/sda1",
"Format": "VMDK",
"DiskImageSize": 570715136.0,
"UserBucket": {
"S3Key": "ami-blogpostvm.ova",
"S3Bucket": "my-import-bucket"
}
}
],
"Platform": "Linux"
}
]
}
</pre>
<h4>
Launch your image</h4>
Your image has now been uploaded and is available as an AMI. Using the AWS CLI, web interface, or any other tool you can launch a new instance and select your AMI.twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0tag:blogger.com,1999:blog-994665793725748583.post-34046752310618884732016-10-01T14:00:00.001+01:002018-12-13T13:19:03.287+00:00Performing nightly build steps with a Jenkinsfile<i><b>Note</b>: 2018-12-13: I have a new <a href="https://hopstorawpointers.blogspot.com/2018/12/nightly-build-steps-with-jenkins.html" target="_blank">post</a> with an updated version that works with declarative pipelines. </i><br />
<br />
<br />
Using a <a href="https://jenkins.io/doc/pipeline/" target="_blank">Jenkinsfile</a> to control your jenkins builds is an important part of the jenkins 2 workflow for pipeline-as-code. A Jenkinsfile allows you to control what you build, were you build it and all other aspects of your CI flow.<br />
<br />
Typically when using pipeline-as-code your build would be triggered by a commit or push from your source control repository. However, there can still be times when you want your build to run on a schedule to perform a long running task e.g. static analysis or a full rebuild of your repository.<br />
<br />
<h4>
Running a nightly build</h4>
Jenkins supports running jobs using a trigger which can be controlled with a cron like format. From a Jenkinsfile this can be setup using triggers <br />
<pre class="brush: ruby">
def triggers = []
triggers << cron('H H(0-2) * * *')
properties (
[
pipelineTriggers(triggers)
]
)
</pre>
This will cause your build to trigger sometime between midnight and 2am every day. The above works correctly, however it will cause a build to trigger for every branch in your repository. To limit it to a specific branch you can change it to<br />
<br />
<br />
<pre class="brush: ruby">def triggers = []
if (env.BRANCH_NAME == "master) {
triggers << cron('H H(0-2) * * *')
}
properties (
[
pipelineTriggers(triggers)
]
)
</pre>
This will limit your scheduled build to only run on the master branch.<br />
<br />
<h4>
Limiting parts of the build to only run at night</h4>
Now that you have your build running every night, how do you limit the long running tasks to only trigger from the nightly build?<br />
<br />
To do this you must examine the cause of the build. This involves getting the rawBuild data and searching all causes for a particular line in the description. Below is a handy function I've written which can be used to get that information.<br />
<br />
// check if the job was started by a timer<br />
<pre class="brush: ruby">// check if the job was started by a timer
@NonCPS
def isJobStartedByTimer() {
def startedByTimer = false
try {
def buildCauses = currentBuild.rawBuild.getCauses()
for ( buildCause in buildCauses ) {
if (buildCause != null) {
def causeDescription = buildCause.getShortDescription()
echo "shortDescription: ${causeDescription}"
if (causeDescription.contains("Started by timer")) {
startedByTimer = true
}
}
}
} catch(theError) {
echo "Error getting build cause"
}
return startedByTimer
}
</pre>
<br />
<blockquote class="tr_bq">
<b>Note</b>: As this is a NonCPS function it must be run outside of a node block.</blockquote>
<blockquote class="tr_bq">
<b>Note</b>: To get this to work correctly you may have to go to Manage Jenkins > In Process Script Approval, and approve the following signatures<br />
<br />
<pre class="brush: ruby">method groovy.lang.Binding getVariables
method hudson.model.Cause getShortDescription
method hudson.model.Run getCause java.lang.Class
method hudson.model.Run getCauses
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
</pre>
</blockquote>
<br />
<br />
When I run my build I change my trigger section to<br />
<br />
<pre class="brush: ruby">def triggers = []
def startedByTimer = false
if (env.BRANCH_NAME == "master) {
triggers << cron('H H(0-2) * * *')
startedByTimer = isJobStartedByTimer()
}
properties (
[
pipelineTriggers(triggers)
]
)
</pre>
<br />
Then later in my build I can check if the build is a timed build and run the additional analysis checks. For example<br />
<br />
<pre class="brush: ruby">if ( startedByTimer ) {
node("analysis_server") {
sh script: "make analysis"
}
}
</pre>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com7tag:blogger.com,1999:blog-994665793725748583.post-54292510589805968522016-06-09T22:06:00.000+01:002016-06-09T22:07:36.851+01:00Multiple Independent Instances of Gnome TerminalMy typical workflow involves SSHing to multiple servers and switching between them. As a result of this I can often end up having 3+ terminals open into 6+ servers. This results in me often having 15+ terminal windows open on top of my usual browsers, file managers, etc.<br />
<br />
I find that it helps me find and sort windows if you can group them based on the server you are logging into instead of the default grouping of all terminals together. To accomplish this grouping you can use a feature in gnome called the window class. This allows you to start applications with a particular WM_CLASS attribute and in the dock, launcher, and <ALT+TAB> menu these windows are grouped together.<br />
<br />
In my previous installations of Ubuntu, I had been using either gnome-shell or xfce as the window manager and xterm as my terminal. With this combination I could easily group my terminals and had a handy script to automatically create a <a href="https://github.com/ttroy50/random-scripts/blob/master/sshmenu/xterm-sshmenu.py" target="_blank">menu launcher.</a> However, after upgrading to Ubuntu 16.04 I decided to investigate using gnome-terminal to replace xterm in my workflow.<br />
<br />
My first attempt was to just change the above script to launch gnome-terminal instead of xterm (with a slight modification of arguments). I quickly found out that this didn't work and some googling told me that the reason is because gnome-terminal launches a background process called gnome-terminal server which in turn launches and controls the terminal windows. I was able to find a <a href="https://chrisirwin.ca/posts/multiple-instances-of-gnome-terminal/" target="_blank">blog</a> on how to launch multiple gnome-terminal-servers, however this required sudo and/or a gnome restart.<br />
<br />
After more investigation I found that in Ubuntu <span style="font-family: "courier new" , "courier" , monospace;">/usr/bin/gnome-terminal</span> is a python script that wraps the startup of gnome-terminal and gnome-terminal-server. With a small change to the script, to add a "--class" flag when launching gnome-terminal-server, I was able to fix the issue of terminal windows not showing in multiple groups. The changed script is available from <a href="https://github.com/ttroy50/random-scripts/blob/master/sshmenu/gnome-terminal-custom" target="_blank">here</a> and the changes are on line 52 and 53 of the script and line 2 and 3 below:<br />
<br />
<pre class="brush: python">
ts = Gio.Subprocess.new(['/usr/lib/gnome-terminal/gnome-terminal-server',
'--class',
name,
'--app-id',
name],
Gio.SubprocessFlags.NONE)
</pre>
Save the script as <span style="font-family: "courier new" , "courier" , monospace;">~/bin/gnome-terminal-custom</span> and to then launch a terminal in it's own class you can call<br />
<pre class="brush: bash, shell">
~/bin/gnome-terminal-custom --disable-factory --app-id com.sshmenu.mylauncher
</pre>
<br />
Wraping up the above in this <a href="https://github.com/ttroy50/random-scripts/blob/master/sshmenu/gt-sshmenu.py" target="_blank">script</a> to create a desktop menu launcher I can easily launch a new terminal that will automatically SSH into a server and group all terminals for that server together.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7eD90PIVVPEzQE2hu3Om9wnaqopc8j31Ite8sMgC-hvFu7IKjF102SvToqd8LPYMRl11oWlsT6C6VQbfaHOfvDgbt0Wpt8XzDwVPSxTa7f-0kWMa1EN_sXPQGd5E668GfSYOUeUOYNu-Q/s1600/desktop-lanucner.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="90" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7eD90PIVVPEzQE2hu3Om9wnaqopc8j31Ite8sMgC-hvFu7IKjF102SvToqd8LPYMRl11oWlsT6C6VQbfaHOfvDgbt0Wpt8XzDwVPSxTa7f-0kWMa1EN_sXPQGd5E668GfSYOUeUOYNu-Q/s320/desktop-lanucner.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Gnome-shell dash showing multiple launchers</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijZsUJ0MRag_HXKbH9koujkHRwY4rQCTQQ3kTjQFOc1yj1PRvW77TG-xbYuqoizefGVGscse7RwYQoDGAJ1xWSelfYmlA49b0CH4icuG1ajKNPSvElr-w_STl_EWolP_B5sFeoc34oV6ia/s1600/dock.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijZsUJ0MRag_HXKbH9koujkHRwY4rQCTQQ3kTjQFOc1yj1PRvW77TG-xbYuqoizefGVGscse7RwYQoDGAJ1xWSelfYmlA49b0CH4icuG1ajKNPSvElr-w_STl_EWolP_B5sFeoc34oV6ia/s320/dock.png" width="118" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">gnome-shell dock showing multiple grouped terminal windows</td></tr>
</tbody></table>
twmatrimhttp://www.blogger.com/profile/08108073888995076828noreply@blogger.com0