Thursday, 5 April 2018

Jenkins build and multi-environment deploys

In our team, we can't run continuous deployment into our testing environments. Like most enterprises with large/legacy back-end systems, we only have a few up-stream instances running with populated data. That results in having a finite set of test environments: qa-1, qa-2, sit-1, sit-2, etc. QA would usually be stubbed out so having one per feature branch wouldn't be a problem but for SIT, having a consistent & known environment for integration testing makes this difficult.

Previously, I used Bamboo and it had the concept of Releases, Environments and dedicated Deploy Jobs. These allowed for builds to be made into releases (manually or automatically) and have those release deployed to specific environments using predefined deploy jobs, with only one release being recorded as deployed to an environment at a time. The advantage of this was it allowed our testers to easily see what was currently deployed and where, without having to dig through every job.

Jenkins 2 was released a while ago and the proposed structure using the pipelines is to have Stages as Deployment Environments. This is great for continuous delivery. However, in a lot of companies, you wouldn't want to immediately push to production (or to other environments) until you were ready. To allow this, it has an input step which will suspend the build until a manual user intervention occurs. This could be used to select which SIT environment you want to deploy to but it has 2 downsides:
  1. The deploy step cannot be re-run once it has been run. So you can't deploy build #2 to sit-1 and sit-2. Or you can't redeploy build #1 to sit-1.
  2. Jobs that haven't been acknowledged aren't marked as done. The jobs remain in a building state. They can have timeouts but that causes more problems.
The middle ground is to have a build job and a deploy jobs for each environment set (qa, sit, staging prod etc) and have these jobs parameterised. Depending on your configuration, you could have just one generic deploy job but there are trade-offs with visibility and safety of accidental production deploys.

In this example project we have a Node.js based AWS Lambda function that we deploy using CloudFront. To test each feature branch before merging, I wanted to have Jenkins automatically to run the tests, create and store a deployable zip and update the pull-request with the build status. Having a stored binary for each build means that the same builds that go through regression testing are the same binaries that get deployed to production, vs doing an npm install, bundling and deploying all in the same job which we were doing previously.

The multi-branch pipeline build

The built zip is published using the S3 Publisher plugin, as we deploy from S3. The regular Jenkins artifact archiver / unarchiver can be used with the same effect if S3 isn't needed. This also uses the NodeJS Plugin to save hassles with nvm. Also AnsiColor and Badge plugins need to be installed.
Here you can see the result of the S3Upload

Build job's Jenkinsfile

This uses the Declarative Pipeline syntax:
pipeline {
  agent {
    node {
      label 'my-node'
  tools { nodejs 'nodejs-6' }
  stages {
    stage('Install') {
      steps {
        // this just makes it a lot easier to view the workspace
        createSummary(icon: 'green.gif', text: "<h1><a href='${JOB_URL}/${BUILD_NUMBER}/execution/node/3/ws/'>Workspace</a></h1>")
        sh 'npm install'
    stage('Build') {
        steps {
          sh 'npm run clean'
          sh "npm run build -- --FILENAME ${FILENAME}"
          s3Upload(entries: [[bucket: "${S3_BUCKET}", sourceFile: "${FILENAME}", selectedRegion: "ap-southeast-2", managedArtifacts: true, flatten: true, uploadFromSlave: true]], profileName: 'S3-Profile', dontWaitForConcurrentBuildCompletion: true, consoleLogLevel: 'INFO', pluginFailureResultConstraint: 'FAILURE', userMetadata: [])
   environment {
      FILENAME = "builds/build_${GIT_BRANCH}_${BUILD_ID}.zip"
      S3_BUCKET = 'my-s3-bucket/releases'
   options {

The deploy job

In the case of the deploy job, I'm kept them as Freestyle projects to make it easy for our testers to go in and modify the different parameters' defaults. This deploy job creates/updates a CloudFormation stack, with the lambda using the file uploaded to S3 before. Unfortunately the S3 Plugin doesn't make this easy to get the url of the artifact programmatically.
The fingerprints create a link between build and deploy jobs
So for this example, create a Freestyle job.

Selecting the branch

  • Make sure the Extensible Choice Plugin "is installed
  • Tick, "This project is parameterized".
    • Add an Extensible Choice parameter,
      Choice Provider: System Groovy Choice Parameter.
    • Set the name to "BRANCH_NAME"
    • Set the following groovy script:
      def parentName = "test-project"
      import jenkins.model.*
      def childJobs(jobName) {
        def parentJob = Jenkins.instance.itemMap.get(jobName)
        return parentJob.getAllJobs()
      return childJobs(parentName).collect {}

A shared script to resolve the S3 url

  • As this script is generic and re-usable, I stored it in "Managed Files".
  • Install the Managed Scripts plugin
  • Go to Manage Jenkins -> Managed Files
  • Click Add a new config
  • Select Groovy file and give it an id of "GetS3ArtifactUrl.groovy"
  • Give it a name of GetS3ArtifactUrl
  • Set the content to:
import hudson.plugins.s3.*
import hudson.model.*
import jenkins.model.*

def lastStableBuild(jobName, branchName) {
    def job = Jenkins.instance.itemMap.get(jobName)
    if (job == null) {
      throw new Exception("error no job found for ${jobName}/${branchName}")
    return job.getLastStableBuild()

def run = lastStableBuild(parentName, branchName)
if (run == null) {
  throw new Exception("error no run found for ${parentName}/${branchName}")

def action = run.getAction(S3ArtifactsAction.class)
def artifact = action.getArtifacts()[0].getArtifact()
if (artifact == null) {
  throw new Exception("error no artifact found for ${parentName}/${branchName}")

def s3_filename="s3://${artifact.bucket}/jobs/${parentName}/${branchName}/${run.number}/${}"

return [S3_FILENAME: s3_filename]

Resolving the S3 URL

  • In the deploy job, under Build Environment, check "Provide Configuration Files".
  • Add file, select the file stored in the shared files. In this case, its "GetS3ArtifactUrl".
  • Fill the target as "GetS3ArtifactUrl.groovy"
  • Make sure the EnvInject plugin is installed
  • Check the "Inject environment variables to the build process" option
    • Set the Groovy Script to:
      binding.parentName = "test-project"
      binding.branchName = "${BRANCH_NAME}"
      return evaluate(new File("${WORKSPACE}/GetS3ArtifactUrl.groovy"))
  • To get the deploy job to pick up the fingerprint of the artifact, the simplest way is to download it using the S3 plugin as it automatically runs the fingerprint against it.
  • Add a build step, "S3 Copy Artifact"
    • Project name: test-project/$BRANCH_NAME
    • Last successful build
    • Stable build only
    • Directory (wherever you like) i.e. tmp/

More limitations

Whilst this does provide a link between the the jobs via Fingerprints, support for linking multi-branch job to other job are blocked by JENKINS-29913 / JENKINS-49197. Currently, Jenkins won't store and show links between the 2 projects. This means you miss out on all the nice dependency graphs and special dependency views etc.

Sunday, 5 November 2017

Using web component polyfills with template tags

I've been playing around with using <template> tags and how well they work with the current Web Component (Custom Elements) polyfills.

My main motivation for going for Web Components instead of something like React or Angular is that I'm currently developing a chrome extension. I wanted the code base to be as small so that it didn't slow down devtools and increase the frequency of hands. Plus I think it's going to be the natural progression from the current React/Angular/etc components - especially with HTTP 2.0's server push of dependant files removing the need for tools like webpack by allow all dependant files to be automatically sent in response to one request.

I immediately hit problems using custom elements in a chrome extension as they're disabled by default. So in order to use them I had to forcefully polyfill the existing API, it took a bit of fiddling  but now works with both libraries I looked at.

Next, using template tags an import link html file, seemed to be creating me a bit of grief. Templates are a key part of making web components easy to build. The contents of a template tag is parsed but it's not considered to be part of the document. This means that when the web components are defined, they are not instantiated as for any tags that are defined inside the template tags until they are attached to the document tree.

There are also 2 types of components:
  • Autonomous custom element
    These are just basically any tag that only extends HTMLElement or a parent class that does. All behaviour and rendering needs to be done by the implementer. They are defined in html as &ltmy-tag>&lt/my-tag>
  • Customized built-in element
    These are components that extend existing elements such as a button, adding to existing functionality. They are defined in html as <button is="my-button"></button>

Importing the elements

In the process of getting the polyfill working in my chrome extension, I came across 2 different ways of adding nodes from an external document. Both were recommend.
This gave me issues with document-register-element which I had to patch to get working until I found the other suggested way of doing it. cloneNode creates a new copy of the node that isn't attached to any document until it is append to a tree.
var link = document.querySelector('link[rel="import"]');
var template = link.import.querySelector('#my-template');
var dest = document.getElementById("insertion-point");

// uses import node
var instance = document.importNode(template.content, true);
Using this made the polyfills behave a bit better. ImportNode creates a copy of the nodes which are attached to the document (but not placed on the tree).
// ....
// uses cloneNode
var instance = template.content.cloneNode(true);

Libraries compared


This is one promoted by polymer / Google's developer site as being the polyfill to use. However I discovered these don't support customized built-in elements. I don't have an immediate need for them but as I get more familiar with using them I'm sure I'll be wanting to use them.
One benefit that this library did have was that when I did a forced polyfill inside the chrome extension's content scripts I was able to use the basic custom elements correctly (but not built-in extensions).


This is a more lightweight implementation and the best to use in other than a chrome extension. Both types of components are supported and the callback methods are called in the right place.
However, using the forced polyfill, the constructors and callback methods were called in the next event loop. This means that at the moment you insert them you can't use them and definitely shouldn't be setting any properties on them as they would overwrite the functionality that hasn't been applied yet.

The comparison

I did a comparison using chrome only as that was my target browser. The files were just locally hosted from my workspace using node express / connect.

Chrome 62.0.3202
customized built-in element
autonomous custom element
Library constructor called createdCallback called connectedCallback called constructor called createdCallback called connectedCallback called
document-register-element.js v1.7.0 After import After constructor After append After import NO After append
document-register-element.js - force polyfill After event loop After constructor After callback After event loop After constructor After callback
webcomponents.js v1.0.17 NO NO NO After import NO After append
webcomponents.js - forced polyfill NO NO NO After import NO After append

So in summary:
  • Use document-register-element for most cases
  • If your forcing the polyfill use webcomponents.js instead at the sacrifice of being able to extend the built-in elements

The test code

<!DOCTYPE html>
        window.module = {}; // for pony version of document-register-element.js
        // uncomment below for forcing webcomponentsjs polyfill
        //if (window.customElements) window.customElements.forcePolyfill = true;
    <script src="/node_modules/@webcomponents/webcomponentsjs/webcomponents-lite.js"></script>

    <!-- enable below for document-register-element -->
    <!--<script src="/node_modules/document-register-element/pony/index.js"></script>-->
      // force polyfill
      //window.module.exports(window, 'force-all');
      // apply with defaults

        class MyButton extends HTMLButtonElement {
            constructor() {
                console.log("MyButton:init  -- customized built-in element");
            createdCallback() {
                console.log("MyButton:createdCallback -- customized built-in element");
                this.textContent = 'button';
            connectedCallback() {
                console.log("MyButton:connectedCallback -- customized built-in element");
            customMethod() {
        // customized built-in element
        customElements.define('my-button', MyButton, {extends: 'button'});

        class MyDiv extends HTMLElement {
           constructor() {
           createdCallback() {
              this.innerHTML = 'button';
           connectedCallback() {
           customMethod() {
        // autonomous custom element
        customElements.define('my-div', MyDiv);
  <!--<link rel="import" href="template-tag-import.html"/>-->

      There should be a "button" text inside the button.
  <div id="insertion-point">

  <template id="my-template">
      <button is="my-button">Something</button>

      var template = document.querySelector('#my-template');
      // if you want to try out the linked document try these statements instead:
      // var link = document.querySelector('link[rel="import"]');
      // var template = link.import.querySelector('#my-template');

      console.log("************* Import node");
      var instance = document.importNode(template.content, true);

      var dest = document.getElementById("insertion-point");
      console.log("************* Appending child");

      window.setTimeout(function() {
        var myButton = dest.querySelector("button");
        console.log("\n\nmyButton has customMethod %s -- customized built-in element", !!myButton.customMethod);

        var myDiv = dest.querySelector("my-div");
        console.log("\n\nmyDiv constructor == MyDiv %s -- autonomous custom element", !!myDiv.customMethod);
      }, 1);

Wednesday, 26 April 2017

How to chain an ES6 Promise

Node.js uses async functions extensively, as it based around non-blocking I/O. Each function takes a callback function parameter, which can result in some messy, deeply nested callback functions if you have to call a bunch of async functions in sequence. Promises make these callbacks a lot cleaner.

ES6 (or ES2016) Promises are a great way of chaining together asynchronous functions so that they read like a series of synchronous statements.

There's already some great posts on Promises, 2ality has a good intro to async then detail of the api, so I won't rehash that article. However, after starting to use them for cases more complicated than most examples, it easy to make a few mistaken assumptions or make things difficult for yourself.

So here is a more complicated example showing a pattern I like to follow. In this example, I'll use the new Javascript Fetch API, which is an API that returns a Promise, allowing you to make async HTTP calls without having to muck around with XMLHttpRequest calls.

First off there are 3 ways to start the chaining, the most obvious one is (taken from MDN, updated to use arrow functions):
function callPromise() {
  return fetch('flowers.jpg');
var myImage = document.querySelector('img');
  .then(response => response.blob())
  .then(myBlob => {
    var objectURL = URL.createObjectURL(myBlob);
    myImage.src = objectURL;
A slightly different:
function callPromise() {
  return fetch('/data');
function handler1(body) {
  console.log("Got body", body);
  .then(response => response.json())
It starts the chain with a promise that immediately calls the next then in the stack. The benefit of this, is that all the statements that perform an action are in the 'then' invocations, so your eye can follow it easier. I personally prefer this way, as I think its to read but both are effectively equivalent.
There is one minor difference that you should be aware of when doing it this way. In the first example callPromise is called immediately when the javascript engine gets to that line. In the 2nd example callPromise is not called until the javascript engine gets to the end of the call stack - it gets called from the event loop.
The 3rd way is by creating a new Promise(). For this article lets just stick to consuming a promise.

The response for each 'then' be any object or undefined which is then passed as the only argument to the next function in the chain. You can also return a promise (or a 'thenable' function) who's final output is used to as the parameter for the next function. So you don't have to resolve any response you return, the promise library automatically normalises this behaviour for you.

A Catch Gotcha 

Once a promise is in a rejected state it will call all 'catch' handlers from that point forward. The 'then' function can take both a onFulfilled and a onRejected parameter and it can be easily mistaken which handlers are called. Looking at the following example, if fetch throws an Error then errorHandler1, errorHandler2 and errorHandler3 will all be called.
    .then(() => fetch('url1'), errorHandler1)
    .then(response => fetch('url2'), errorHandler2)
    .then(response => fetch('url3'), errorHandeler3);
So how do you achieve what you were intending if you did the above? The answer is to add the errorHandler to the promises returned in each of the fulfilled handlers before they get added to the outer promise chain. An example exlpains it a lot better, applying it to the above:
    .then(() => {
        return fetch('url1')
    .then(response => {
        return fetch('url2')
    .then(response => {
        return fetch('url3')
In the above, only one the errorHandlers 1-3 only get called if the individual fetch call fails. If one of the fetch fails then the finalErrorHandler is called, so you could use it as a single place to return an error response up the async stack.

Continuing a Promise After an Error

Usually if a Promise chain goes into an error state, it will call all the error handlers and never call any more of the fulfilled handlers. If you want the fulfilled handlers to be continued to be called when the error can be recovered from, then you need to return a Promise in a fulfilled state.
    .then(() => {
        throw new Error();
    .catch(err => {
        // handle recoverable error
        return Promise.resolve(); // returns a Promise in a fulfilled state
    .then(() => {
        // this handler will be called

Making Testable Promises

Using arrow functions in promises makes for pretty code in blog posts but if the promises are implemented this way it makes them hard to test. In order to test each handler function, you have to call the entire chain, which for anything other than the most trival chain is tedious and makes for bad unit tests. So to make them testable, my recommendation is to not use arrow functions or function expressions in a promise chain. Instead create a named function for each handler that is exported so it can be invoked from a test case. There are cases where it make sense to use a one line arrow function but use sparingly. I'll post a complete example in another post.

Using a Promise to Once-only Init

Promises can only settle once, calling the attached stack once it resolves only once. Additionally, calling 'then' or 'catch' on an already settled promise, will immediately can the passed function. This can make it handy to implement asynchronous init events, for instance a AWS Lambda function that queries DynamoDB as soon as it boots up and then finishes the initialisation on the the first request that it handles using environment variables stored in API Gateway.
import aws  from 'aws-sdk';
import SpotifyWebApi from 'spotify-web-api-node';
const dynamoDB = new aws.DynamoDB.DocumentClient({region: 'ap-southeast-2'});

var apiLoadedResolve;
var apiLoadedPromise = new Promise((resolve, reject) => {
    apiLoadedResolve = resolve;

export function lambdaHandler(request, lambdaContext, callback) {
    let api = getApi(request);
    // handle individual request
    return {}; // response object

var spotifyApi;
function getApi(request) {
    if (spotifyApi)
        return spotifyApi;

    spotifyApi = new SpotifyWebApi({
        clientId: request.env.SPOTIFY_CLIENT_ID,
        clientSecret: request.env.SPOTIFY_SECRET,        
    // this only triggers the promise to be settled once

    return spotifyApi;

// access dynamo db on script load before lambdaHandler is called
        TableName: "TestTable",
        Key: {name: "initData"}
    .then(item => {
        let conf = item.Item;
        apiLoadedPromise = apiLoadedPromise.then(spotifyApi => {
            // passed in on first request

This implementation doesn't require the config from the database to be set before being able to process requests. If you needed the config to be set before processing requests add a conditional check to see if it's init'ed, if not execute the rest of the function as a then callback to apiLoadedPromise

Saturday, 15 April 2017

Adding MDC headers to every Spring MVC request

Mapped Diagnostic Context (MDC) logging allows you to set attributes associated with current thread, so that SLF4J (via your logger implementation library) can log those attributes with each logging statement without it having to be specified.

For example you could configure logback to log the sessionId on every statement, which is really handy when using a log indexer such as Splunk. This would allow you to easily see all the requests made by a user, for a given session. To use with logback, you'd set the pattern to %-4r [%thread] %-5level sessionId=%X{sessionId} - %msg%n

Setting these attributes for each entry point would be a pain so one way would be to implement a ServletRequestListener, which would allow setting the attributes at the start of the request and removing them again at the end of the request. (Note: It's important to remove the attributes afterwards, as threads are re-used by application servers and will give misleading logging statements)
If you're using Spring MVC then an alternative is to implement a HandlerInterceptor. How to configure it using annotations instead of XML is not immediately obvious, so here it is so I don't have to work it out again next time I need it.

This implementation just pulls values out of the request object without much manipulation. If you want to add the method parameters of an annotated handler method then you'll need to use Spring AOP instead.

HandlerInterceptor Implementation

import org.slf4j.MDC;
import org.springframework.web.servlet.handler.HandlerInterceptorAdapter;

import java.util.HashSet;
import java.util.Set;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

 * Adds logging.
public class LoggingHandlerInterceptor extends HandlerInterceptorAdapter {
    * set of keys added to MDC so can be removed
   private ThreadLocal<Set<String>> storedKeys = ThreadLocal.withInitial(() -> new HashSet<>());

   public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
      addKey("sessionId", request.getHeader("X-Session-Id"));
      addKey("url", request.getRequestURI());
      if (request.getHeader("X-Request-Id") != null) {
         addKey("requestId", request.getHeader("X-Request-Id"));
      return true;

   private void addKey(String key, String value) {
      MDC.put(key, value);

   public void afterConcurrentHandlingStarted(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
      // request ended on current thread remove properties

   public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex)
           throws Exception {

   private void removeKeys() {
      for (String key : storedKeys.get()) {

Spring Java Annotation Configuration

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;

public class BeanConfiguration {

   public LoggingHandlerInterceptor loggingHandlerInterceptor() {
      return new LoggingHandlerInterceptor();

   public WebMvcConfigurerAdapter webConfigurer() {
      return new WebMvcConfigurerAdapter() {
         public void addInterceptors(InterceptorRegistry registry) {

And if your using logback and Spring Boot here is the configuration to output all MDC keys (using spring's default formatting):


<?xml version="1.0" encoding="UTF-8"?>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <property name="CONSOLE_LOG_PATTERN" value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m %mdc%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
  <property name="FILE_LOG_PATTERN" value="%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %m %mdc%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
  <include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>

Tuesday, 14 March 2017

Populating stored procs into a HSQL DB

I recently encountered a problem trying to load stored procedures into a HSQL DB used for testing. The problem was caused by the script runner provided by spring which separates each statement to be executed in a script file by a semicolon. If a stored proc has statements inside it (which most do), then the proc isn't executed as a single statement. This is further compounded by each statement executed must be understandable by JDBC. For example the following stored proc causes issues:
    SELECT the_value INTO out_param FROM my_table WHERE field = param1;
This problem is solved by using the script runners provided by HSQL in the "org.hsqldb:sqltool" dependency as they parse can correctly parse the scripts containing stored procedures. Here is a Spring Boot test, using an in memory database but using HSQL's script runners:
public class MyDaoTest {

  private MyDao dao;

  public void myTest() throws Exception {
    String response = dao.invokeProc("1234");
    assertThat(response, notNullValue());

  public static class Config {

    public EmbeddedDatabaseFactoryBean dataSource() {
      EmbeddedDatabaseFactoryBean factory = new EmbeddedDatabaseFactoryBean();
      return factory;

    public HsqlDbPopulator databasePopulator(String... scripts) {
      return new HsqlDbPopulator(scripts);

  public static class HsqlDbPopulator implements DatabasePopulator {
    private final String[] scriptFiles;

    public HsqlDbPopulator(String[] scripts) {
      this.scriptFiles = scripts;

    public void populate(Connection connection) throws SQLException, ScriptException {
      FileSystemResourceLoader resourceLoader = new FileSystemResourceLoader();
      for (String scriptFile : scriptFiles) {
        try {
          SqlFile file = new SqlFile(resourceLoader.getResource(scriptFile).getFile(), null, false);

"Running script {}", scriptFile);
        } catch (IOException | SqlToolError e) {
          log.error("Error executing script {}", scriptFile, e);
          throw new UncategorizedScriptException("Error executing script " + scriptFile, e);
Note: this test uses an in memory database, if you want to use another variation, you'll need to create a custom EmbeddedDatabaseConfigurer instance to do this which is slightly painful due to Spring's insistence on making everything either private or final or both.