Wednesday, 26 April 2017

How to chain an ES6 Promise

Node.js uses async functions extensively, as it based around non-blocking I/O. Each function takes a callback function parameter, which can result in some messy, deeply nested callback functions if you have to call a bunch of async functions in sequence. Promises make these callbacks a lot cleaner.

ES6 (or ES2016) Promises are a great way of chaining together asynchronous functions so that they read like a series of synchronous statements.

There's already some great posts on Promises, 2ality has a good intro to async then detail of the api, so I won't rehash that article. However, after starting to use them for cases more complicated than most examples, it easy to make a few mistaken assumptions or make things difficult for yourself.

So here is a more complicated example showing a pattern I like to follow. In this example, I'll use the new Javascript Fetch API, which is an API that returns a Promise, allowing you to make async HTTP calls without having to muck around with XMLHttpRequest calls.

First off there are 3 ways to start the chaining, the most obvious one is (taken from MDN, updated to use arrow functions):
function callPromise() {
  return fetch('flowers.jpg');
var myImage = document.querySelector('img');
  .then(response => response.blob())
  .then(myBlob => {
    var objectURL = URL.createObjectURL(myBlob);
    myImage.src = objectURL;
A slightly different:
function callPromise() {
  return fetch('/data');
function handler1(body) {
  console.log("Got body", body);
  .then(response => response.json())
It starts the chain with a promise that immediately calls the next then in the stack. The benefit of this, is that all the statements that perform an action are in the 'then' invocations, so your eye can follow it easier. I personally prefer this way, as I think its to read but both are effectively equivalent.
There is one minor difference that you should be aware of when doing it this way. In the first example callPromise is called immediately when the javascript engine gets to that line. In the 2nd example callPromise is not called until the javascript engine gets to the end of the call stack - it gets called from the event loop.
The 3rd way is by creating a new Promise(). For this article lets just stick to consuming a promise.

The response for each 'then' be any object or undefined which is then passed as the only argument to the next function in the chain. You can also return a promise (or a 'thenable' function) who's final output is used to as the parameter for the next function. So you don't have to resolve any response you return, the promise library automatically normalises this behaviour for you.

A Catch Gotcha 

Once a promise is in a rejected state it will call all 'catch' handlers from that point forward. The 'then' function can take both a onFulfilled and a onRejected parameter and it can be easily mistaken which handlers are called. Looking at the following example, if fetch throws an Error then errorHandler1, errorHandler2 and errorHandler3 will all be called.
    .then(() => fetch('url1'), errorHandler1)
    .then(response => fetch('url2'), errorHandler2)
    .then(response => fetch('url3'), errorHandeler3);
So how do you achieve what you were intending if you did the above? The answer is to add the errorHandler to the promises returned in each of the fulfilled handlers before they get added to the outer promise chain. An example exlpains it a lot better, applying it to the above:
    .then(() => {
        return fetch('url1')
    .then(response => {
        return fetch('url2')
    .then(response => {
        return fetch('url3')
In the above, only one the errorHandlers 1-3 only get called if the individual fetch call fails. If one of the fetch fails then the finalErrorHandler is called, so you could use it as a single place to return an error response up the async stack.

Continuing a Promise After an Error

Usually if a Promise chain goes into an error state, it will call all the error handlers and never call any more of the fulfilled handlers. If you want the fulfilled handlers to be continued to be called when the error can be recovered from, then you need to return a Promise in a fulfilled state.
    .then(() => {
        throw new Error();
    .catch(err => {
        // handle recoverable error
        return Promise.resolve(); // returns a Promise in a fulfilled state
    .then(() => {
        // this handler will be called

Making Testable Promises

Using arrow functions in promises makes for pretty code in blog posts but if the promises are implemented this way it makes them hard to test. In order to test each handler function, you have to call the entire chain, which for anything other than the most trival chain is tedious and makes for bad unit tests. So to make them testable, my recommendation is to not use arrow functions or function expressions in a promise chain. Instead create a named function for each handler that is exported so it can be invoked from a test case. There are cases where it make sense to use a one line arrow function but use sparingly. I'll post a complete example in another post.

Using a Promise to Once-only Init

Promises can only settle once, calling the attached stack once it resolves only once. Additionally, calling 'then' or 'catch' on an already settled promise, will immediately can the passed function. This can make it handy to implement asynchronous init events, for instance a AWS Lambda function that queries DynamoDB as soon as it boots up and then finishes the initialisation on the the first request that it handles using environment variables stored in API Gateway.
import aws  from 'aws-sdk';
import SpotifyWebApi from 'spotify-web-api-node';
const dynamoDB = new aws.DynamoDB.DocumentClient({region: 'ap-southeast-2'});

var apiLoadedResolve;
var apiLoadedPromise = new Promise((resolve, reject) => {
    apiLoadedResolve = resolve;

export function lambdaHandler(request, lambdaContext, callback) {
    let api = getApi(request);
    // handle individual request
    return {}; // response object

var spotifyApi;
function getApi(request) {
    if (spotifyApi)
        return spotifyApi;

    spotifyApi = new SpotifyWebApi({
        clientId: request.env.SPOTIFY_CLIENT_ID,
        clientSecret: request.env.SPOTIFY_SECRET,        
    // this only triggers the promise to be settled once

    return spotifyApi;

// access dynamo db on script load before lambdaHandler is called
        TableName: "TestTable",
        Key: {name: "initData"}
    .then(item => {
        let conf = item.Item;
        apiLoadedPromise = apiLoadedPromise.then(spotifyApi => {
            // passed in on first request

This implementation doesn't require the config from the database to be set before being able to process requests. If you needed the config to be set before processing requests add a conditional check to see if it's init'ed, if not execute the rest of the function as a then callback to apiLoadedPromise

Saturday, 15 April 2017

Adding MDC headers to every Spring MVC request

Mapped Diagnostic Context (MDC) logging allows you to set attributes associated with current thread, so that SLF4J (via your logger implementation library) can log those attributes with each logging statement without it having to be specified.

For example you could configure logback to log the sessionId on every statement, which is really handy when using a log indexer such as Splunk. This would allow you to easily see all the requests made by a user, for a given session. To use with logback, you'd set the pattern to %-4r [%thread] %-5level sessionId=%X{sessionId} - %msg%n

Setting these attributes for each entry point would be a pain so one way would be to implement a ServletRequestListener, which would allow setting the attributes at the start of the request and removing them again at the end of the request. (Note: It's important to remove the attributes afterwards, as threads are re-used by application servers and will give misleading logging statements)
If you're using Spring MVC then an alternative is to implement a HandlerInterceptor. How to configure it using annotations instead of XML is not immediately obvious, so here it is so I don't have to work it out again next time I need it.

This implementation just pulls values out of the request object without much manipulation. If you want to add the method parameters of an annotated handler method then you'll need to use Spring AOP instead.

HandlerInterceptor Implementation

import org.slf4j.MDC;
import org.springframework.web.servlet.handler.HandlerInterceptorAdapter;

import java.util.HashSet;
import java.util.Set;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

 * Adds logging.
public class LoggingHandlerInterceptor extends HandlerInterceptorAdapter {
    * set of keys added to MDC so can be removed
   private ThreadLocal<Set<String>> storedKeys = ThreadLocal.withInitial(() -> new HashSet<>());

   public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
      addKey("sessionId", request.getHeader("X-Session-Id"));
      addKey("url", request.getRequestURI());
      if (request.getHeader("X-Request-Id") != null) {
         addKey("requestId", request.getHeader("X-Request-Id"));
      return true;

   private void addKey(String key, String value) {
      MDC.put(key, value);

   public void afterConcurrentHandlingStarted(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
      // request ended on current thread remove properties

   public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex)
           throws Exception {

   private void removeKeys() {
      for (String key : storedKeys.get()) {

Spring Java Annotation Configuration

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;

public class BeanConfiguration {

   public LoggingHandlerInterceptor loggingHandlerInterceptor() {
      return new LoggingHandlerInterceptor();

   public WebMvcConfigurerAdapter webConfigurer() {
      return new WebMvcConfigurerAdapter() {
         public void addInterceptors(InterceptorRegistry registry) {

And if your using logback and Spring Boot here is the configuration to output all MDC keys (using spring's default formatting):


<?xml version="1.0" encoding="UTF-8"?>
  <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
  <property name="CONSOLE_LOG_PATTERN" value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m %mdc%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
  <property name="FILE_LOG_PATTERN" value="%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %m %mdc%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>
  <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
  <include resource="org/springframework/boot/logging/logback/file-appender.xml"/>
  <root level="INFO">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>

Tuesday, 14 March 2017

Populating stored procs into a HSQL DB

I recently encountered a problem trying to load stored procedures into a HSQL DB used for testing. The problem was caused by the script runner provided by spring which separates each statement to be executed in a script file by a semicolon. If a stored proc has statements inside it (which most do), then the proc isn't executed as a single statement. This is further compounded by each statement executed must be understandable by JDBC. For example the following stored proc causes issues:
    SELECT the_value INTO out_param FROM my_table WHERE field = param1;
This problem is solved by using the script runners provided by HSQL in the "org.hsqldb:sqltool" dependency as they parse can correctly parse the scripts containing stored procedures. Here is a Spring Boot test, using an in memory database but using HSQL's script runners:
public class MyDaoTest {

  private MyDao dao;

  public void myTest() throws Exception {
    String response = dao.invokeProc("1234");
    assertThat(response, notNullValue());

  public static class Config {

    public EmbeddedDatabaseFactoryBean dataSource() {
      EmbeddedDatabaseFactoryBean factory = new EmbeddedDatabaseFactoryBean();
      return factory;

    public HsqlDbPopulator databasePopulator(String... scripts) {
      return new HsqlDbPopulator(scripts);

  public static class HsqlDbPopulator implements DatabasePopulator {
    private final String[] scriptFiles;

    public HsqlDbPopulator(String[] scripts) {
      this.scriptFiles = scripts;

    public void populate(Connection connection) throws SQLException, ScriptException {
      FileSystemResourceLoader resourceLoader = new FileSystemResourceLoader();
      for (String scriptFile : scriptFiles) {
        try {
          SqlFile file = new SqlFile(resourceLoader.getResource(scriptFile).getFile(), null, false);

"Running script {}", scriptFile);
        } catch (IOException | SqlToolError e) {
          log.error("Error executing script {}", scriptFile, e);
          throw new UncategorizedScriptException("Error executing script " + scriptFile, e);
Note: this test uses an in memory database, if you want to use another variation, you'll need to create a custom EmbeddedDatabaseConfigurer instance to do this which is slightly painful due to Spring's insistence on making everything either private or final or both.

Friday, 10 February 2017

Dev Setup of a Mac

After working in my 2nd consecutive company that uses Mac's for developers and having forgot everything I used the first time, I thought I better write down all the tweaks, work arounds and config changes that got me using my mac efficiently.

Remap Fn-C to copy

Mac's use Mac-C instead of Ctrl-C to copy (and X, V to cut and paste). This is really annoying if you switch between a mac and Windows machine a lot. Fortunately you can use Karabiner to map Fn-C to copy as the Fn key is located where Ctrl is on a windows keyboard. If your on OSX Sierra you'll need to use Karabiner Elements for now.


This needs to be installed first as it installs most dev tools.
/usr/bin/ruby -e "$(curl -fsSL"

A better terminal

brew install iterm2

Colour ls 

From this Stackexchange post, edit ~/.bash_profile:
export CLICOLOR=1
export LSCOLORS= gxBxhxDxfxhxhxhxhxcxcx

Colour vim 

edit ~/.vimrc:
syntax on

AWS etc

awslogs is a cloudwatch log watcher (you may have to run the install as sudo)

brew install python
brew install awscli
pip install awslogs


sshuttle - a handy python ssh tunnelling that allows you to easily route ranges of IPs over ssh.

pip install sshuttle


brew install rebenv
Add the following to ~/bash_profile
eval "$(rbenv init -)"
Then run
rbenv install 2.2.3
rbenv global 2.2.3


brew cask install java
brew install gradle
Then add this to ~/.bash_profile:
export JAVA_HOME=$(/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home)

Sunday, 29 November 2015

Running a meteor shell on a standalone server

For those that want to connect to meteor's shell running on a standalone/self-maintained server, the standard command you use in the development environment doesn't work. Fortunately you can 'trick' it into allowing the `meteor shell` command.
export METEOR_SHELL_DIR="$APPDIR/.meteor/local/shell"
# other settings
# ...

exec node $APPDIR/bundle/main.js
cd /opt/bitnami/apps/myapp
mkdir -p .meteor/local/shell
echo > .meteor/packages
echo 'METEOR@1.2.1\n' > .meteor/release

meteor shell